You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/09 13:25:16 UTC

[GitHub] [arrow] vibhatha opened a new pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation [WIP]

vibhatha opened a new pull request #12590:
URL: https://github.com/apache/arrow/pull/12590


   WIP draft PR for Scalar UDF integration


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839515132



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -199,6 +202,135 @@ FunctionDoc = namedtuple(
      "options_required"))
 
 
+cdef wrap_arity(const CArity c_arity):
+    """
+    Wrap a C++ Arity in an Arity object
+    """
+    cdef Arity arity = Arity.__new__(Arity)
+    arity.init(c_arity)
+    return arity
+
+
+cdef wrap_input_type(const CInputType c_input_type):
+    """
+    Wrap a C++ InputType in an InputType object
+    """
+    cdef InputType input_type = InputType.__new__(InputType)
+    input_type.init(c_input_type)
+    return input_type
+
+
+cdef class InputType(_Weakrefable):
+    """
+    An interface for defining input-types for streaming execution engine
+    applications. 
+    """
+
+    def __init__(self):
+        raise TypeError("Cannot use constructor to initialize InputType")
+
+    cdef void init(self, const CInputType &input_type):
+        self.input_type = input_type
+
+    @staticmethod
+    def scalar(data_type):
+        """
+        create a scalar input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.scalar(pa.int32())
+        <pyarrow._compute.InputType object at 0x1029fdcb0>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Scalar(c_data_type)
+        return wrap_input_type(c_input_type)
+
+    @staticmethod
+    def array(data_type):
+        """
+        create an array input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.array(pa.int32())
+        <pyarrow._compute.InputType object at 0x102ba4850>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Array(c_data_type)
+        return wrap_input_type(c_input_type)
+
+
+cdef class Arity(_Weakrefable):

Review comment:
       I am not sure if I understood the suggestion correctly. The current code outputs the following,
   
   ```python
   pc.get_function("py_add_one_func")
   ```
   
   ```bash
   arrow.compute.Function<name=py_add_one_func, kind=scalar, arity=1, num_kernels=1>
   ```
   
   I also understand we can just use the interface defined in the `libarrow.pxd` directly without using this class. 
   Is it what is intended in this suggestion? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1084979289


   This is for function registration so I don't think there will be any dynamic content here.  For example, we currently have "scalar" and we know we will need "aggregate".  There may be other classes of UDF that we add but each time we do so it will be intentional an accompanied by a code change.  I don't think we're looking into dynamically adding new classes of UDFs.
   
   Function options & state are a slightly different story.  We do want to support dynamic function option content.  However, from the C++ perspective, both of these things will just be `PyObject*`.  For example, maybe a user defines a custom datetime formatting UDF and they want to take the format pattern and locale in as objects.  I think we could do it this way...
   
   ```
   class UdfOptions(object):
     def __init__(self):
       pass
   
   class CustomDateFormatOptions(UdfOptions):
     def __init__(self, format, locale):
       self.format = format
       self.locale = locale
   ```
   
   Function registration remains unchanged.  Later, when they call their UDF we would do something like...
   
   ```
   pc.call_function("custom_date_format", [timestamps_arr], CustomDateFormatOptions(my_format, my_locale))
   ```
   
   Then the python layer could check to see if the options object extends `UdfOptions` and, if it does, pass it to some kind of `CUdfOptions`:
   
   ```
   public struct UdfOptions {
     PyObject* options_obj;
   };
   ```
   
   ...then we'd probably need some logic when we are actually calling the PyFunc to grab the PyObject out of the options so that hopefully the UDF itself could look something like...
   
   ```
   def custom_date_format(arr, options):
     ...
   ```
   In summary, I think a flat struct should be sufficient for any cases we need to tackle.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831735697



##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       Yes indeed. I also added a note just in-case it is not related. Will add in a minor PR. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831742324



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)
+        )
+    in_types: List[InputType]
+        list of InputType objects which defines the input 
+        types for the function
+    out_type: DataType
+        output type of the function
+    callback: callable
+        user defined function 
+    mem_allocation: str
+        memory allocation mode 
+        "preallocate" or "no_preallocate"

Review comment:
       Yes, this is pretty unclear, I will enhance the docs. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1084498299


   > @vibhatha I'm not entirely clear on the concern. I don't mind if we have an inheritence tree of object structs. For example, this would be ok (this is totally made up and likely is not correct for aggregate UDFs at all):
   > 
   > ```
   > struct UdfOptions {
   >   cp::Function::Kind kind;
   >   cp::Arity arity;
   >   const cp::FunctionDoc func_doc;
   >   std::vector<cp::InputType> in_types;
   >   cp::OutputType out_type;
   >   cp::MemAllocation::type mem_allocation;
   >   cp::NullHandling::type null_handling;
   > };
   > 
   > struct ScalarUdfOptions : public UdfOptions {
   >   std::string func_name;
   > };
   > 
   > struct AggregateUdfOptions : public UdfOptions {
   >   std::string init_func_name;
   >   std::string map_func_name;
   >   std::string reduce_func_name;
   > };
   > 
   > Status MakeScalarFunction(PyObject* function, const ScalarUdfOptions& options);
   > Status MakeAggregateFunction(PyObject* function, const AggregateUdfOptions& options);
   > ```
   
   @westonpace This make sense to me. How about if we have to go for dynamic content, where we have to attach those values from Python to this struct. I haven't explored this deeply, that's why I mentioned that part is yet to be implemented. I am not 100% sure if we need such an option, but yet to verify it's necessity. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831741825



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)

Review comment:
       I understand, that's better. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831742101



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)
+        )
+    in_types: List[InputType]
+        list of InputType objects which defines the input 
+        types for the function
+    out_type: DataType
+        output type of the function
+    callback: callable
+        user defined function 

Review comment:
       Sure, I can extend the docs-example. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace edited a comment on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
westonpace edited a comment on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1084197255


   @vibhatha I'm not entirely clear on the concern.  I don't mind if we have an inheritence tree of object structs.  For example, this would be ok (this is totally made up and likely is not correct for aggregate UDFs at all):
   
   ```
   struct UdfOptions {
     cp::Function::Kind kind;
     cp::Arity arity;
     const cp::FunctionDoc func_doc;
     std::vector<cp::InputType> in_types;
     cp::OutputType out_type;
     cp::MemAllocation::type mem_allocation;
     cp::NullHandling::type null_handling;
   };
   
   struct ScalarUdfOptions : public UdfOptions {
     std::string func_name;
   };
   
   struct AggregateUdfOptions : public UdfOptions {
     std::string init_func_name;
     std::string map_func_name;
     std::string reduce_func_name;
   };
   
   Status MakeScalarFunction(PyObject* function, const ScalarUdfOptions& options);
   Status MakeAggregateFunction(PyObject* function, const AggregateUdfOptions& options);
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r841158856



##########
File path: python/pyarrow/tests/test_udf.py
##########
@@ -0,0 +1,350 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import List
+
+import pytest
+
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str]):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    return func_doc
+
+# scalar unary function data
+
+
+unary_doc = get_function_doc("add function",
+                             "test add function",
+                             ["scalar1"])
+
+
+def unary_function(scalar1):
+    return pc.call_function("add", [scalar1, 1])
+
+# scalar binary function data
+
+
+binary_doc = get_function_doc("y=mx",
+                              "find y from y = mx",
+                              ["m", "x"])
+
+
+def binary_function(m, x):
+    return pc.call_function("multiply", [m, x])
+
+# scalar ternary function data
+
+
+ternary_doc = get_function_doc("y=mx+c",
+                               "find y from y = mx + c",
+                               ["m", "x", "c"])
+
+
+def ternary_function(m, x, c):
+    mx = pc.call_function("multiply", [m, x])
+    return pc.call_function("add", [mx, c])
+
+# scalar varargs function data
+
+
+varargs_doc = get_function_doc("z=ax+by+c",
+                               "find z from z = ax + by + c",
+                               ["a", "x", "b", "y", "c"])
+
+
+def varargs_function(a, x, b, y, c):
+    ax = pc.call_function("multiply", [a, x])
+    by = pc.call_function("multiply", [b, y])
+    ax_by = pc.call_function("add", [ax, by])
+    return pc.call_function("add", [ax_by, c])
+
+
+@pytest.fixture
+def function_input_types():
+    return [
+        # scalar data input types
+        [
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        # array data input types
+        [
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ]
+    ]
+
+
+@pytest.fixture
+def function_output_types():
+    return [
+        pa.int64(),
+        pa.int64(),
+        pa.int64(),
+        pa.int64()
+    ]
+
+
+@pytest.fixture
+def function_names():
+    return [
+        # scalar data function names
+        "scalar_y=x+k",
+        "scalar_y=mx",
+        "scalar_y=mx+c",
+        "scalar_z=ax+by+c",
+        # array data function names
+        "array_y=x+k",
+        "array_y=mx",
+        "array_y=mx+c",
+        "array_z=ax+by+c"
+    ]
+
+
+@pytest.fixture
+def function_arities():
+    return [
+        1,
+        2,
+        3,
+        5,
+    ]
+
+
+@pytest.fixture
+def function_docs():
+    return [
+        unary_doc,
+        binary_doc,
+        ternary_doc,
+        varargs_doc
+    ]
+
+
+@pytest.fixture
+def functions():
+    return [
+        unary_function,
+        binary_function,
+        ternary_function,
+        varargs_function
+    ]
+
+
+@pytest.fixture
+def function_inputs():
+    return [
+        # scalar input data
+        [
+            pa.scalar(10, pa.int64())
+        ],
+        [
+            pa.scalar(10, pa.int64()),
+            pa.scalar(2, pa.int64())
+        ],
+        [
+            pa.scalar(10, pa.int64()),
+            pa.scalar(2, pa.int64()),
+            pa.scalar(5, pa.int64())
+        ],
+        [
+            pa.scalar(2, pa.int64()),
+            pa.scalar(10, pa.int64()),
+            pa.scalar(3, pa.int64()),
+            pa.scalar(20, pa.int64()),
+            pa.scalar(5, pa.int64())
+        ],
+        # array input data
+        [
+            pa.array([10, 20], pa.int64())
+        ],
+        [
+            pa.array([10, 20], pa.int64()),
+            pa.array([2, 4], pa.int64())
+        ],
+        [
+            pa.array([10, 20], pa.int64()),
+            pa.array([2, 4], pa.int64()),
+            pa.array([5, 10], pa.int64())
+        ],
+        [
+            pa.array([2, 3], pa.int64()),
+            pa.array([10, 20], pa.int64()),
+            pa.array([3, 7], pa.int64()),
+            pa.array([20, 30], pa.int64()),
+            pa.array([5, 10], pa.int64())
+        ]
+    ]
+
+
+@pytest.fixture
+def expected_outputs():
+    return [
+        # scalar output data
+        pa.scalar(11, pa.int64()),  # 10 + 1
+        pa.scalar(20, pa.int64()),  # 10 * 2
+        pa.scalar(25, pa.int64()),  # 10 * 2 + 5
+        pa.scalar(85, pa.int64()),  # (2 * 10) + (3 * 20) + 5
+        # array output data
+        pa.array([11, 21], pa.int64()),  # [10 + 1, 20 + 1]
+        pa.array([20, 80], pa.int64()),  # [10 * 2, 20 * 4]
+        pa.array([25, 90], pa.int64()),  # [(10 * 2) + 5, (20 * 4) + 10]
+        # [(2 * 10) + (3 * 20) + 5, (3 * 20) + (7 * 30) + 10]
+        pa.array([85, 280], pa.int64())
+    ]
+
+
+def test_scalar_udf_function_with_scalar_data(function_names,

Review comment:
       On the contrary, I am thinking is it wise to create another PR for unregister and multi-kernel regisration functionality to keep this PR simpler and easy to review. @westonpace WDYT? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839296023



##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       Yes, that's right. I removed it. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840995527



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }
+  if (!PyCallable_Check(function_.obj())) {
+    return Status::TypeError("Expected a callable python object.");
+  }
+  auto doc = options->doc();
+  auto arity = options->arity();
+  scalar_func_ = std::make_shared<compute::ScalarFunction>(options->name(), arity, &doc);
+
+  // lambda function
+  auto call_back = [&, arity](compute::KernelContext* ctx,

Review comment:
       I understand. We can switch to this. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840995466



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }
+  if (!PyCallable_Check(function_.obj())) {
+    return Status::TypeError("Expected a callable python object.");
+  }
+  auto doc = options->doc();
+  auto arity = options->arity();
+  scalar_func_ = std::make_shared<compute::ScalarFunction>(options->name(), arity, &doc);

Review comment:
       Yes that's right. I have missed this. Thanks a lot for spotting this one. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831740573



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -199,6 +202,135 @@ FunctionDoc = namedtuple(
      "options_required"))
 
 
+cdef wrap_arity(const CArity c_arity):
+    """
+    Wrap a C++ Arity in an Arity object
+    """
+    cdef Arity arity = Arity.__new__(Arity)
+    arity.init(c_arity)
+    return arity
+
+
+cdef wrap_input_type(const CInputType c_input_type):
+    """
+    Wrap a C++ InputType in an InputType object
+    """
+    cdef InputType input_type = InputType.__new__(InputType)
+    input_type.init(c_input_type)
+    return input_type
+
+
+cdef class InputType(_Weakrefable):
+    """
+    An interface for defining input-types for streaming execution engine
+    applications. 
+    """
+
+    def __init__(self):
+        raise TypeError("Cannot use constructor to initialize InputType")
+
+    cdef void init(self, const CInputType &input_type):
+        self.input_type = input_type
+
+    @staticmethod
+    def scalar(data_type):
+        """
+        create a scalar input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.scalar(pa.int32())
+        <pyarrow._compute.InputType object at 0x1029fdcb0>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Scalar(c_data_type)
+        return wrap_input_type(c_input_type)
+
+    @staticmethod
+    def array(data_type):
+        """
+        create an array input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.array(pa.int32())
+        <pyarrow._compute.InputType object at 0x102ba4850>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Array(c_data_type)
+        return wrap_input_type(c_input_type)
+
+
+cdef class Arity(_Weakrefable):

Review comment:
       Yes indeed we can digest it as `num_of_args` and do the object mapping internally. 
   It could be more Pythonic than this right? I think you're correct as this is the case in https://www.sqlite.org/c3ref/create_function.html. 
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1074693774


   @westonpace thank you for the detailed review. I will work on the suggested changes. It is exciting to write this new feature. Appreciate your support. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839332923



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }
+    if (PyCallable_Check(py_function)) {
+      RETURN_NOT_OK(VerifyArityAndInput(arity, batch));
+      if (batch[0].is_array()) {  // checke 0-th element to select array callable

Review comment:
       I didn’t get this clearly. The Python function expects an array or scalar as the input. So we have to do the conversion at some point, am I wrong here? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840995247



##########
File path: python/pyarrow/tests/test_udf.py
##########
@@ -0,0 +1,350 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import List
+
+import pytest
+
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str]):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    return func_doc
+
+# scalar unary function data
+
+
+unary_doc = get_function_doc("add function",
+                             "test add function",
+                             ["scalar1"])
+
+
+def unary_function(scalar1):
+    return pc.call_function("add", [scalar1, 1])
+
+# scalar binary function data
+
+
+binary_doc = get_function_doc("y=mx",
+                              "find y from y = mx",
+                              ["m", "x"])
+
+
+def binary_function(m, x):
+    return pc.call_function("multiply", [m, x])
+
+# scalar ternary function data
+
+
+ternary_doc = get_function_doc("y=mx+c",
+                               "find y from y = mx + c",
+                               ["m", "x", "c"])
+
+
+def ternary_function(m, x, c):
+    mx = pc.call_function("multiply", [m, x])
+    return pc.call_function("add", [mx, c])
+
+# scalar varargs function data
+
+
+varargs_doc = get_function_doc("z=ax+by+c",
+                               "find z from z = ax + by + c",
+                               ["a", "x", "b", "y", "c"])
+
+
+def varargs_function(a, x, b, y, c):
+    ax = pc.call_function("multiply", [a, x])
+    by = pc.call_function("multiply", [b, y])
+    ax_by = pc.call_function("add", [ax, by])
+    return pc.call_function("add", [ax_by, c])
+
+
+@pytest.fixture
+def function_input_types():
+    return [
+        # scalar data input types
+        [
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        # array data input types
+        [
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ]
+    ]
+
+
+@pytest.fixture
+def function_output_types():
+    return [
+        pa.int64(),
+        pa.int64(),
+        pa.int64(),
+        pa.int64()
+    ]
+
+
+@pytest.fixture
+def function_names():
+    return [
+        # scalar data function names
+        "scalar_y=x+k",
+        "scalar_y=mx",
+        "scalar_y=mx+c",
+        "scalar_z=ax+by+c",
+        # array data function names
+        "array_y=x+k",
+        "array_y=mx",
+        "array_y=mx+c",
+        "array_z=ax+by+c"
+    ]
+
+
+@pytest.fixture
+def function_arities():
+    return [
+        1,
+        2,
+        3,
+        5,
+    ]
+
+
+@pytest.fixture
+def function_docs():
+    return [
+        unary_doc,
+        binary_doc,
+        ternary_doc,
+        varargs_doc
+    ]
+
+
+@pytest.fixture
+def functions():
+    return [
+        unary_function,
+        binary_function,
+        ternary_function,
+        varargs_function
+    ]
+
+
+@pytest.fixture
+def function_inputs():
+    return [
+        # scalar input data
+        [
+            pa.scalar(10, pa.int64())
+        ],
+        [
+            pa.scalar(10, pa.int64()),
+            pa.scalar(2, pa.int64())
+        ],
+        [
+            pa.scalar(10, pa.int64()),
+            pa.scalar(2, pa.int64()),
+            pa.scalar(5, pa.int64())
+        ],
+        [
+            pa.scalar(2, pa.int64()),
+            pa.scalar(10, pa.int64()),
+            pa.scalar(3, pa.int64()),
+            pa.scalar(20, pa.int64()),
+            pa.scalar(5, pa.int64())
+        ],
+        # array input data
+        [
+            pa.array([10, 20], pa.int64())
+        ],
+        [
+            pa.array([10, 20], pa.int64()),
+            pa.array([2, 4], pa.int64())
+        ],
+        [
+            pa.array([10, 20], pa.int64()),
+            pa.array([2, 4], pa.int64()),
+            pa.array([5, 10], pa.int64())
+        ],
+        [
+            pa.array([2, 3], pa.int64()),
+            pa.array([10, 20], pa.int64()),
+            pa.array([3, 7], pa.int64()),
+            pa.array([20, 30], pa.int64()),
+            pa.array([5, 10], pa.int64())
+        ]
+    ]
+
+
+@pytest.fixture
+def expected_outputs():
+    return [
+        # scalar output data
+        pa.scalar(11, pa.int64()),  # 10 + 1
+        pa.scalar(20, pa.int64()),  # 10 * 2
+        pa.scalar(25, pa.int64()),  # 10 * 2 + 5
+        pa.scalar(85, pa.int64()),  # (2 * 10) + (3 * 20) + 5
+        # array output data
+        pa.array([11, 21], pa.int64()),  # [10 + 1, 20 + 1]
+        pa.array([20, 80], pa.int64()),  # [10 * 2, 20 * 4]
+        pa.array([25, 90], pa.int64()),  # [(10 * 2) + 5, (20 * 4) + 10]
+        # [(2 * 10) + (3 * 20) + 5, (3 * 20) + (7 * 30) + 10]
+        pa.array([85, 280], pa.int64())
+    ]
+
+
+def test_scalar_udf_function_with_scalar_data(function_names,

Review comment:
       It will be better to fix it in this PR itself since it is a core functionality. Along with this, I also thought about how to provide access to register a function which can have multiple input, output configurations. Simply we can take a list of such functions from the user and add multiple kernels when we register the function. It seems to be the most straightforward way to solve this problem. Other way is to create a hash with the name and input and output types, so even though the user calls a function by name, internally we infer the types and get the registered name. But it could be overly complicating things. Would like to know your thoughts on this. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation [WIP]

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1062923344






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831738300



##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DECLARE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)               \
+  ARROW_PYTHON_EXPORT Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch&, \
+                                                             PyObject*, int, Datum*);
+
+DECLARE_CALL_UDF(Scalar, scalar, scalar)
+DECLARE_CALL_UDF(Array, array, make_array)
+
+#undef DECLARE_CALL_UDF
+
+class ARROW_PYTHON_EXPORT UdfBuilder {

Review comment:
       Sure, I we can add that option. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r836100352



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)

Review comment:
       Created a JIRA here: https://issues.apache.org/jira/browse/ARROW-16041
   This will be integrated eventually. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r836068466



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions&, const cp::FunctionOptions&) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(const cp::FunctionOptions&) const override;

Review comment:
       It is implemented below. 
   
   ```c++
   std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
       const cp::FunctionOptions&) const {
     return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1084163268


   @pitrou regarding this: https://github.com/apache/arrow/pull/12590#discussion_r837509180
   
   Ah I see your point. But a few things about this. 
   
   Aren't we are going to user the options feature to take in UDF specific options. The listed attributes in this suggestion are core to any UDF. I haven't deeply thought about how to expose dynamic options to Python yet. That's why it was empty in the first place. What do you think about this? 
   
   cc @westonpace 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r837630807



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {

Review comment:
       We can do that too. For now I will keep it simple, since we are not yet exposing options to UDFs. At least not in this PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1082049454


   @pitrou Thanks a lot for the very descriptive review. I will address these issues. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r837484456



##########
File path: cpp/examples/arrow/CMakeLists.txt
##########
@@ -133,4 +133,11 @@ if(ARROW_PARQUET AND ARROW_DATASET)
 
   add_arrow_example(join_example EXTRA_LINK_LIBS ${DATASET_EXAMPLES_LINK_LIBS})
   add_dependencies(join-example parquet)
+
+  add_arrow_example(udf_example EXTRA_LINK_LIBS ${DATASET_EXAMPLES_LINK_LIBS})
+  add_dependencies(udf-example parquet)
+
+  add_arrow_example(aggregate_example EXTRA_LINK_LIBS ${DATASET_EXAMPLES_LINK_LIBS})
+  add_dependencies(aggregate-example parquet)

Review comment:
       Why the Parquet dependencies?

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;

Review comment:
       Can we reserve all-caps for macros?
   ```suggestion
     using ArrowArrayType = typename arrow::TypeTraits<TYPE>::ArrayType;
     using ArrowBuilderType = typename arrow::TypeTraits<TYPE>::BuilderType;
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {

Review comment:
       Can we give this a meaningful name? For example "TernarySumOptionsType"?

##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace cp = arrow::compute;

Review comment:
       It's not recommended to declare namespace aliases in `.h` files, they might crash with third-party code.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,

Review comment:
       Nit: we know it's a lambda, no need to suffix it with "lambda"...

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }
+    if (PyCallable_Check(py_function)) {

Review comment:
       Same here: check this on construction.

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;

Review comment:
       ```suggestion
     return arrow::Table::Make(schema, data_vector, /*num_rows=*/6);
   ```

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2182,3 +2322,257 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if func_doc and isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        # UDFOptions integration:
+        # TODO: https://issues.apache.org/jira/browse/ARROW-16041
+        f_doc.options_class = tobytes("None")
+
+        c_options_required = False
+        f_doc.options_required = c_options_required
+
+        return f_doc
+    else:
+        raise ValueError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):

Review comment:
       I don't think it's useful to define custom exceptions here, just use built-in ones.

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -199,6 +202,135 @@ FunctionDoc = namedtuple(
      "options_required"))
 
 
+cdef wrap_arity(const CArity c_arity):
+    """
+    Wrap a C++ Arity in an Arity object
+    """
+    cdef Arity arity = Arity.__new__(Arity)
+    arity.init(c_arity)
+    return arity
+
+
+cdef wrap_input_type(const CInputType c_input_type):
+    """
+    Wrap a C++ InputType in an InputType object
+    """
+    cdef InputType input_type = InputType.__new__(InputType)
+    input_type.init(c_input_type)
+    return input_type
+
+
+cdef class InputType(_Weakrefable):
+    """
+    An interface for defining input-types for streaming execution engine
+    applications. 
+    """
+
+    def __init__(self):
+        raise TypeError("Cannot use constructor to initialize InputType")
+
+    cdef void init(self, const CInputType &input_type):
+        self.input_type = input_type
+
+    @staticmethod
+    def scalar(data_type):
+        """
+        create a scalar input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.scalar(pa.int32())
+        <pyarrow._compute.InputType object at 0x1029fdcb0>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Scalar(c_data_type)
+        return wrap_input_type(c_input_type)
+
+    @staticmethod
+    def array(data_type):
+        """
+        create an array input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.array(pa.int32())
+        <pyarrow._compute.InputType object at 0x102ba4850>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Array(c_data_type)
+        return wrap_input_type(c_input_type)
+
+
+cdef class Arity(_Weakrefable):

Review comment:
       This doesn't seem resolved to me. I agree this custom class is completely overkill. Also, it doesn't match how this information is currently exposed:
   ```python
   >>> import pyarrow as pa, pyarrow.compute as pc
   >>> pc.get_function("random")
   arrow.compute.Function<name=random, kind=scalar, arity=0, num_kernels=1>
   >>> pc.get_function("sort_indices")
   arrow.compute.Function<name=sort_indices, kind=meta, arity=1, num_kernels=0>
   >>> pc.get_function("add")
   arrow.compute.Function<name=add, kind=scalar, arity=2, num_kernels=33>
   >>> pc.get_function("if_else")
   arrow.compute.Function<name=if_else, kind=scalar, arity=3, num_kernels=39>
   >>> pc.get_function("choose")
   arrow.compute.Function<name=choose, kind=scalar, arity=Ellipsis, num_kernels=32>
   ```
   (note the varargs convention of returning Ellipsis is perhaps not enough if we want to faithfully mirror the C++ function metadata, but that can be a followup PR)
   

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {

Review comment:
       It's not very useful that the custom options class doesn't carry any actual option. Is it useful? Do we want to add something there to spice things up?

##########
File path: python/pyarrow/public-api.pxi
##########
@@ -38,7 +37,6 @@ cdef api shared_ptr[CBuffer] pyarrow_unwrap_buffer(object buffer):
 
     return shared_ptr[CBuffer]()
 
-

Review comment:
       Why these changes?

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;

Review comment:
       Can instead write:
   ```suggestion
     ARROW_RETURN_NOT_OK(builder.AppendValues(values));
     return builder.Finish();
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {
+ public:
+  ExampleNode(ExecNode* input, const ExampleNodeOptions&)
+      : ExecNode(/*plan=*/input->plan(), /*inputs=*/{input},
+                 /*input_labels=*/{"ignored"},
+                 /*output_schema=*/input->output_schema(), /*num_outputs=*/1) {}
+
+  const char* kind_name() const override { return "ExampleNode"; }
+
+  arrow::Status StartProducing() override {
+    outputs_[0]->InputFinished(this, 0);
+    return arrow::Status::OK();
+  }
+
+  void ResumeProducing(ExecNode* output) override {}
+  void PauseProducing(ExecNode* output) override {}
+
+  void StopProducing(ExecNode* output) override { inputs_[0]->StopProducing(this); }
+  void StopProducing() override { inputs_[0]->StopProducing(); }
+
+  void InputReceived(ExecNode* input, cp::ExecBatch batch) override {}
+  void ErrorReceived(ExecNode* input, arrow::Status error) override {}
+  void InputFinished(ExecNode* input, int total_batches) override {}
+
+  arrow::Future<> finished() override { return inputs_[0]->finished(); }
+};
+
+arrow::Result<cp::ExecNode*> ExampleExecNodeFactory(cp::ExecPlan* plan,
+                                                    std::vector<cp::ExecNode*> inputs,
+                                                    const cp::ExecNodeOptions& options) {
+  const auto& example_options =
+      arrow::internal::checked_cast<const ExampleNodeOptions&>(options);
+
+  return plan->EmplaceNode<ExampleNode>(inputs[0], example_options);
+}
+
+const cp::FunctionDoc func_doc{
+    "User-defined-function usage to demonstrate registering an out-of-tree function",
+    "returns x + y + z",
+    {"x", "y", "z"},
+    "UDFOptions"};
+
+arrow::Status SampleFunction(cp::KernelContext* ctx, const cp::ExecBatch& batch,
+                             arrow::Datum* out) {
+  auto in_res = cp::CallFunction("add", {batch[0].array(), batch[1].array()});
+  auto in_arr = in_res.ValueOrDie().make_array();
+  auto final_res = cp::CallFunction("add", {in_arr, batch[2].array()});
+  auto final_arr = final_res.ValueOrDie().array();
+  auto datum = new arrow::Datum(final_arr);
+  *out = *datum;
+  return arrow::Status::OK();
+}
+
+arrow::Status Execute() {
+  const std::string name = "add_three";
+  auto func = std::make_shared<cp::ScalarFunction>(name, cp::Arity::Ternary(), &func_doc);
+
+  auto options = std::make_shared<UDFOptions>();
+  cp::ScalarKernel kernel(
+      {cp::InputType::Array(arrow::int64()), cp::InputType::Array(arrow::int64()),
+       cp::InputType::Array(arrow::int64())},
+      arrow::int64(), SampleFunction);
+
+  kernel.mem_allocation = cp::MemAllocation::NO_PREALLOCATE;
+  kernel.null_handling = cp::NullHandling::COMPUTED_NO_PREALLOCATE;
+
+  ABORT_ON_FAILURE(func->AddKernel(std::move(kernel)));

Review comment:
       This function is returning a `Status`, so you can use `ARROW_RETURN_NOT_OK` everywhere.

##########
File path: cpp/src/arrow/python/api.h
##########
@@ -28,3 +28,4 @@
 #include "arrow/python/numpy_to_arrow.h"
 #include "arrow/python/python_to_arrow.h"
 #include "arrow/python/serialize.h"
+#include "arrow/python/udf.h"

Review comment:
       I don't really think it makes sense to add this. `udf.h` is just an internal helper to be used from our Cython code, right?

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {
+ public:
+  ExampleNode(ExecNode* input, const ExampleNodeOptions&)
+      : ExecNode(/*plan=*/input->plan(), /*inputs=*/{input},
+                 /*input_labels=*/{"ignored"},
+                 /*output_schema=*/input->output_schema(), /*num_outputs=*/1) {}
+
+  const char* kind_name() const override { return "ExampleNode"; }
+
+  arrow::Status StartProducing() override {
+    outputs_[0]->InputFinished(this, 0);
+    return arrow::Status::OK();
+  }
+
+  void ResumeProducing(ExecNode* output) override {}
+  void PauseProducing(ExecNode* output) override {}
+
+  void StopProducing(ExecNode* output) override { inputs_[0]->StopProducing(this); }
+  void StopProducing() override { inputs_[0]->StopProducing(); }
+
+  void InputReceived(ExecNode* input, cp::ExecBatch batch) override {}
+  void ErrorReceived(ExecNode* input, arrow::Status error) override {}
+  void InputFinished(ExecNode* input, int total_batches) override {}
+
+  arrow::Future<> finished() override { return inputs_[0]->finished(); }
+};
+
+arrow::Result<cp::ExecNode*> ExampleExecNodeFactory(cp::ExecPlan* plan,
+                                                    std::vector<cp::ExecNode*> inputs,
+                                                    const cp::ExecNodeOptions& options) {
+  const auto& example_options =
+      arrow::internal::checked_cast<const ExampleNodeOptions&>(options);
+
+  return plan->EmplaceNode<ExampleNode>(inputs[0], example_options);
+}
+
+const cp::FunctionDoc func_doc{
+    "User-defined-function usage to demonstrate registering an out-of-tree function",
+    "returns x + y + z",
+    {"x", "y", "z"},
+    "UDFOptions"};
+
+arrow::Status SampleFunction(cp::KernelContext* ctx, const cp::ExecBatch& batch,
+                             arrow::Datum* out) {
+  auto in_res = cp::CallFunction("add", {batch[0].array(), batch[1].array()});
+  auto in_arr = in_res.ValueOrDie().make_array();
+  auto final_res = cp::CallFunction("add", {in_arr, batch[2].array()});
+  auto final_arr = final_res.ValueOrDie().array();
+  auto datum = new arrow::Datum(final_arr);
+  *out = *datum;
+  return arrow::Status::OK();

Review comment:
       We should promote proper error handling in examples.
   Untested suggestion:
   ```suggestion
     // temp = x + y; return temp + z
     ARROW_ASSIGN_OR_RAISE(auto temp, cp::CallFunction("add", {batch[0], batch[1]}));
     ARROW_ASSIGN_OR_RAISE(*out, cp::CallFunction("add", {temp, batch[2]}));
     return arrow::Status::OK();
   ```

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }

Review comment:
       This should be checked above while constructing the scalar function.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }
+    if (PyCallable_Check(py_function)) {
+      RETURN_NOT_OK(VerifyArityAndInput(arity, batch));
+      if (batch[0].is_array()) {  // checke 0-th element to select array callable

Review comment:
       I don't understand why you have two different branches here. You could just wrap the objects underlying the input Datums, regardless of their kind.

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {

Review comment:
       I'm curious: why does a UDF function example need a custom ExecNode implementation?

##########
File path: python/pyarrow/_compute.pxd
##########
@@ -21,6 +21,19 @@ from pyarrow.lib cimport *
 from pyarrow.includes.common cimport *
 from pyarrow.includes.libarrow cimport *
 
+cdef class Arity(_Weakrefable):

Review comment:
       Note that declaring those classes here is only useful if they are being used from other Cython modules, which doesn't seem to be the case.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();

Review comment:
       Our style guide mandates C++-style casts, so `static_cast` here.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;

Review comment:
       You need to use `OwnedRefNoGIL` so that the object is actually kept alive by the lambda function.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }
+    if (PyCallable_Check(py_function)) {
+      RETURN_NOT_OK(VerifyArityAndInput(arity, batch));
+      if (batch[0].is_array()) {  // checke 0-th element to select array callable
+        RETURN_NOT_OK(exec_function_array(batch, py_function, arity.num_args, out));
+      } else if (batch[0].is_scalar()) {  // check 0-th element to select scalar callable
+        RETURN_NOT_OK(exec_function_scalar(batch, py_function, arity.num_args, out));
+      } else {
+        return Status::Invalid("Unexpected input type, scalar or array type expected.");
+      }
+    } else {
+      return Status::ExecutionError("Expected a callable python object.");
+    }
+    return Status::OK();
+  };  // lambda function
+
+  cp::ScalarKernel kernel(
+      cp::KernelSignature::Make(this->input_types(), this->output_type(),
+                                this->arity().is_varargs),
+      call_back_lambda);
+  kernel.mem_allocation = this->mem_allocation();
+  kernel.null_handling = this->null_handling();
+  st = func->AddKernel(std::move(kernel));
+  if (!st.ok()) {
+    return Status::ExecutionError("Kernel couldn't be added to the udf : " +
+                                  st.message());
+  }

Review comment:
       Let's just propagate the existing error instead of making another one:
   ```suggestion
     RETURN_NOT_OK(func->AddKernel(std::move(kernel)));
   ```

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }
+    if (PyCallable_Check(py_function)) {
+      RETURN_NOT_OK(VerifyArityAndInput(arity, batch));
+      if (batch[0].is_array()) {  // checke 0-th element to select array callable
+        RETURN_NOT_OK(exec_function_array(batch, py_function, arity.num_args, out));
+      } else if (batch[0].is_scalar()) {  // check 0-th element to select scalar callable
+        RETURN_NOT_OK(exec_function_scalar(batch, py_function, arity.num_args, out));
+      } else {
+        return Status::Invalid("Unexpected input type, scalar or array type expected.");
+      }
+    } else {
+      return Status::ExecutionError("Expected a callable python object.");
+    }
+    return Status::OK();
+  };  // lambda function
+
+  cp::ScalarKernel kernel(
+      cp::KernelSignature::Make(this->input_types(), this->output_type(),
+                                this->arity().is_varargs),
+      call_back_lambda);
+  kernel.mem_allocation = this->mem_allocation();
+  kernel.null_handling = this->null_handling();
+  st = func->AddKernel(std::move(kernel));
+  if (!st.ok()) {
+    return Status::ExecutionError("Kernel couldn't be added to the udf : " +
+                                  st.message());
+  }
+  auto registry = cp::GetFunctionRegistry();
+  st = registry->AddFunction(std::move(func));

Review comment:
       Same here.

##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DECLARE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)               \
+  ARROW_PYTHON_EXPORT Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch&, \
+                                                             PyObject*, int, Datum*);
+
+DECLARE_CALL_UDF(Scalar, scalar, scalar)
+DECLARE_CALL_UDF(Array, array, make_array)
+
+#undef DECLARE_CALL_UDF
+
+class ARROW_PYTHON_EXPORT UdfBuilder {

Review comment:
       @vibhatha Hmm, I think you misunderstood @westonpace 's suggestion. AFAIU he was suggesting something like this:
   ```c++
   namespace arrow {
   namespace py {
   namespace compute {
   
   struct UdfOptions {
     std::string func_name;
     cp::Function::Kind kind;
     cp::Arity arity;
     const cp::FunctionDoc func_doc;
     std::vector<cp::InputType> in_types;
     cp::OutputType out_type;
     cp::MemAllocation::type mem_allocation;
     cp::NullHandling::type null_handling;
   };
   
   Status MakeScalarFunction(PyObject* function, const UdfOptions& options);
   ```

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;
+  auto arity = this->arity();
+  // lambda function
+  auto call_back_lambda = [py_function, arity](cp::KernelContext* ctx,
+                                               const cp::ExecBatch& batch,
+                                               Datum* out) -> Status {
+    PyAcquireGIL lock;
+    if (py_function == NULL) {
+      return Status::ExecutionError("python function cannot be null");
+    }
+    if (PyCallable_Check(py_function)) {
+      RETURN_NOT_OK(VerifyArityAndInput(arity, batch));
+      if (batch[0].is_array()) {  // checke 0-th element to select array callable
+        RETURN_NOT_OK(exec_function_array(batch, py_function, arity.num_args, out));
+      } else if (batch[0].is_scalar()) {  // check 0-th element to select scalar callable
+        RETURN_NOT_OK(exec_function_scalar(batch, py_function, arity.num_args, out));
+      } else {
+        return Status::Invalid("Unexpected input type, scalar or array type expected.");
+      }
+    } else {
+      return Status::ExecutionError("Expected a callable python object.");

Review comment:
       Please use `TypeError` here.

##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,352 @@
+from typing import List

Review comment:
       Can we move the examples to a separate PR and JIRA? It will make this PR a bit less long to review :-)

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \

Review comment:
       Indeed, please let's avoid macros that are more than 2 or 3 lines long.

##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       So this file should be removed, right?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840921085



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,

Review comment:
       Also just a general comment, please be cognizant of using CamelCase




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831736619



##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This example showcases various ways to work with Datasets. It's
+// intended to be paired with the documentation.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+namespace cp = arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {arrow::field("a", arrow::int64()),
+                       arrow::field("b", arrow::boolean()),
+                       arrow::field("c", arrow::int64())};
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({0, 1, 2, 0, 4, 1, 0, 5}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array,
+                        GetArrayDataSample<arrow::BooleanType>(
+                            {false, true, false, true, true, false, true, false}));
+  ARROW_ASSIGN_OR_RAISE(auto data_array, GetArrayDataSample<arrow::Int64Type>(
+                                             {10, 11, 12, 10, 11, 11, 10, 15}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, bool_array, data_array};
+
+  table = arrow::Table::Make(schema, data_vector, 8);
+
+  return table;
+}
+
+arrow::Status DoAggregate() {
+  auto maybe_plan = cp::ExecPlan::Make();
+  ABORT_ON_FAILURE(maybe_plan.status());
+  auto plan = maybe_plan.ValueOrDie();
+  cp::ExecContext exec_context(arrow::default_memory_pool(),
+                               ::arrow::internal::GetCpuThreadPool());
+
+  ARROW_ASSIGN_OR_RAISE(auto table, GetTable());
+
+  std::cout << "Source Table" << std::endl;
+
+  std::cout << table->ToString() << std::endl;
+
+  std::shared_ptr<arrow::Table> out;
+  cp::CountOptions options(cp::CountOptions::ONLY_VALID);
+  auto aggregate_options = cp::AggregateNodeOptions{/*aggregates=*/{{"sum", &options}},
+                                                    /*targets=*/{"c"},
+                                                    /*names=*/{"count(c)"},
+                                                    /*keys=*/{}};
+  auto schema = arrow::schema({arrow::field("count(c)", arrow::int64())});
+
+  ABORT_ON_FAILURE(cp::Declaration::Sequence(
+                       {
+                           {"table_source", cp::TableSourceNodeOptions{table, 2}},
+                           {"aggregate", aggregate_options},
+                           {"table_sink", cp::TableSinkNodeOptions{&out, schema}},
+                       })
+                       .AddToPlan(plan.get())
+                       .status());
+
+  ARROW_RETURN_NOT_OK(plan->StartProducing());
+
+  std::cout << "Output Table Data : " << std::endl;
+  std::cout << out->ToString() << std::endl;
+
+  auto future = plan->finished();
+
+  return future.status();
+}
+
+int main(int argc, char** argv) {

Review comment:
       Yes, that's a good point. I thought that aggregations could be a little more interesting topic for many users. That's why I wrote a separate one. I can extend this one further with a good use case, if that is a better contribution than the current one. And can do a separate PR for that. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831741292



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 

Review comment:
       That's a good question. Then we need to somehow create a unique signature for that before we call the C++ interface. 
   I see your point and it is very usable if we can provide that. So the arity, in_types can be used to generate that unique function name. Is this is a correct understanding?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831738103



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \

Review comment:
       I see your point. I also thought about this a few times, I ended up writing almost the same code for the two cases that's why I thought of simplying it. It's hard to read and debug as well. I definitely can fallback to the original version of that function implementation. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831742456



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)
+        )
+    in_types: List[InputType]
+        list of InputType objects which defines the input 
+        types for the function
+    out_type: DataType
+        output type of the function
+    callback: callable
+        user defined function 
+    mem_allocation: str
+        memory allocation mode 
+        "preallocate" or "no_preallocate"
+    null_handling: str
+        null handling mode
+        one of "intersect", "computed_preallocate",
+        "computed_no_preallocate", 
+        "output_not_null"

Review comment:
       Yes, this needs further explanation in the docs for each possible option. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r836056283



##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This example showcases various ways to work with Datasets. It's
+// intended to be paired with the documentation.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+namespace cp = arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {arrow::field("a", arrow::int64()),
+                       arrow::field("b", arrow::boolean()),
+                       arrow::field("c", arrow::int64())};
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({0, 1, 2, 0, 4, 1, 0, 5}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array,
+                        GetArrayDataSample<arrow::BooleanType>(
+                            {false, true, false, true, true, false, true, false}));
+  ARROW_ASSIGN_OR_RAISE(auto data_array, GetArrayDataSample<arrow::Int64Type>(
+                                             {10, 11, 12, 10, 11, 11, 10, 15}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, bool_array, data_array};
+
+  table = arrow::Table::Make(schema, data_vector, 8);
+
+  return table;
+}
+
+arrow::Status DoAggregate() {
+  auto maybe_plan = cp::ExecPlan::Make();
+  ABORT_ON_FAILURE(maybe_plan.status());
+  auto plan = maybe_plan.ValueOrDie();
+  cp::ExecContext exec_context(arrow::default_memory_pool(),
+                               ::arrow::internal::GetCpuThreadPool());
+
+  ARROW_ASSIGN_OR_RAISE(auto table, GetTable());
+
+  std::cout << "Source Table" << std::endl;
+
+  std::cout << table->ToString() << std::endl;
+
+  std::shared_ptr<arrow::Table> out;
+  cp::CountOptions options(cp::CountOptions::ONLY_VALID);
+  auto aggregate_options = cp::AggregateNodeOptions{/*aggregates=*/{{"sum", &options}},
+                                                    /*targets=*/{"c"},
+                                                    /*names=*/{"count(c)"},
+                                                    /*keys=*/{}};
+  auto schema = arrow::schema({arrow::field("count(c)", arrow::int64())});
+
+  ABORT_ON_FAILURE(cp::Declaration::Sequence(
+                       {
+                           {"table_source", cp::TableSourceNodeOptions{table, 2}},
+                           {"aggregate", aggregate_options},
+                           {"table_sink", cp::TableSinkNodeOptions{&out, schema}},
+                       })
+                       .AddToPlan(plan.get())
+                       .status());
+
+  ARROW_RETURN_NOT_OK(plan->StartProducing());
+
+  std::cout << "Output Table Data : " << std::endl;
+  std::cout << out->ToString() << std::endl;
+
+  auto future = plan->finished();
+
+  return future.status();
+}
+
+int main(int argc, char** argv) {

Review comment:
       Another motivation is to extend this with the UDF integration to aggregations. May be it is too early to include it in this PR. May be we can exclude it for now. What do you think? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839213447



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {

Review comment:
       Removing this from the current example. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839213300



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {

Review comment:
       Removing this component from the example and using it with default options. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r837628569



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {

Review comment:
       Not necessarily, it’s just to show case how the function is used with an execution engine application. I see your point, we can be more objective and just stick to a function example. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839211160



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>  // ARROW-15263
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/make_unique.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// Demonstrate registering a user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions& options,
+               const cp::FunctionOptions& other) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(
+      const cp::FunctionOptions& options) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {

Review comment:
       removing this content and simplifying the example. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840882112



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();

Review comment:
       ```suggestion
     ARROW_ASSIGN_OR_RAISE(auto unwrapped_result, unwrap_scalar(result));
     *out = unwrapped_result;
     return Status::OK();
   ```
   It may be a little odd you can do `*out = unwrapped_result` where `out` is a `Datum` and `unwrapped_result` is a `Scalar` but `Datum` was intentionally designed with a lot of implicit constructors to allow for [`implicit conversion`](https://en.cppreference.com/w/cpp/language/implicit_conversion)
   
   If you really want to be concise...we have a pattern for "set output parameter from result and return status":
   ```
   return unwrap_scalar(result).Value(out);
   ```
   
   How it is written right now is actually a memory leak.  We use `shared_ptr` and `unique_ptr` almost exclusively for lifecycle management in Arrow.  Any time you are doing a `new` (except for `std::unique_ptr<Foo>(new Foo(...))` it should be a red flag to double check that you really do intend for this.
   
   When a function has an out parameter (`Datum * out`) you know that the caller has already allocated a spot for this (possibly on the stack, possibly on the heap) so there is no need to do any allocation here.  All you need to do is fill in the spot the caller has allocated.
   
   You could do...
   ```
   auto datum = new Datum(c_res_data);
   *out = *datum;
   delete datum;
   ```
   but that would just be both complex and inefficient.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }
+  if (!PyCallable_Check(function_.obj())) {
+    return Status::TypeError("Expected a callable python object.");
+  }
+  auto doc = options->doc();
+  auto arity = options->arity();
+  scalar_func_ = std::make_shared<compute::ScalarFunction>(options->name(), arity, &doc);
+
+  // lambda function
+  auto call_back = [&, arity](compute::KernelContext* ctx,

Review comment:
       Nit: Normally we write `callback` without the space.
   
   I'm not sure callback is a good name though.  I think of a callback as something that runs when a task is finished.  This is the kernel's exec function.  Can we just call it `exec`?

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();

Review comment:
       ```suggestion
     return unwrap_array(result).Value(out);
   ```

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;

Review comment:
       It's a bit odd to have this defined so far away from where it is used.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");

Review comment:
       This error message seems incomplete.

##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace arrow {
+
+namespace py {
+
+// Exposing the UDFOptions: https://issues.apache.org/jira/browse/ARROW-16041
+class ARROW_PYTHON_EXPORT UdfOptions {
+ public:
+  UdfOptions(const compute::Function::Kind kind, const compute::Arity arity,
+             const compute::FunctionDoc func_doc,
+             const std::vector<compute::InputType> in_types,
+             const compute::OutputType out_type,
+             const compute::MemAllocation::type mem_allocation,
+             const compute::NullHandling::type null_handling)
+      : kind_(kind),
+        arity_(arity),
+        func_doc_(func_doc),
+        in_types_(in_types),
+        out_type_(out_type),
+        mem_allocation_(mem_allocation),
+        null_handling_(null_handling) {}
+
+  compute::Function::Kind kind() { return kind_; }
+
+  const compute::Arity& arity() const { return arity_; }
+
+  const compute::FunctionDoc doc() const { return func_doc_; }
+
+  const std::vector<compute::InputType>& input_types() const { return in_types_; }
+
+  const compute::OutputType& output_type() const { return out_type_; }
+
+  compute::MemAllocation::type mem_allocation() { return mem_allocation_; }

Review comment:
       I'm not sure it makes sense to expose `mem_allocation` to the user.  If I understand correctly, using `PREALLOCATE` means that the `out` datum passed to the kernel function is already populated with allocated buffers.  So the kernel function can just fill in the buffers instead of allocating new buffers.  Right now there is no way for a python function to fill in preallocated buffers and I can't imagine that is something we are going to need anytime soon.
   
   So for this PR I would hide `mem_allocation` from the user entirely and just hard code it to `NO_PREALLOCATE`.

##########
File path: python/pyarrow/tests/test_udf.py
##########
@@ -0,0 +1,350 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import List
+
+import pytest
+
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str]):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    return func_doc
+
+# scalar unary function data
+
+
+unary_doc = get_function_doc("add function",
+                             "test add function",
+                             ["scalar1"])
+
+
+def unary_function(scalar1):
+    return pc.call_function("add", [scalar1, 1])
+
+# scalar binary function data
+
+
+binary_doc = get_function_doc("y=mx",
+                              "find y from y = mx",
+                              ["m", "x"])
+
+
+def binary_function(m, x):
+    return pc.call_function("multiply", [m, x])
+
+# scalar ternary function data
+
+
+ternary_doc = get_function_doc("y=mx+c",
+                               "find y from y = mx + c",
+                               ["m", "x", "c"])
+
+
+def ternary_function(m, x, c):
+    mx = pc.call_function("multiply", [m, x])
+    return pc.call_function("add", [mx, c])
+
+# scalar varargs function data
+
+
+varargs_doc = get_function_doc("z=ax+by+c",
+                               "find z from z = ax + by + c",
+                               ["a", "x", "b", "y", "c"])
+
+
+def varargs_function(a, x, b, y, c):
+    ax = pc.call_function("multiply", [a, x])
+    by = pc.call_function("multiply", [b, y])
+    ax_by = pc.call_function("add", [ax, by])
+    return pc.call_function("add", [ax_by, c])
+
+
+@pytest.fixture
+def function_input_types():
+    return [
+        # scalar data input types
+        [
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        [
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64()),
+            InputType.scalar(pa.int64())
+        ],
+        # array data input types
+        [
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ],
+        [
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64()),
+            InputType.array(pa.int64())
+        ]
+    ]
+
+
+@pytest.fixture
+def function_output_types():
+    return [
+        pa.int64(),
+        pa.int64(),
+        pa.int64(),
+        pa.int64()
+    ]
+
+
+@pytest.fixture
+def function_names():
+    return [
+        # scalar data function names
+        "scalar_y=x+k",
+        "scalar_y=mx",
+        "scalar_y=mx+c",
+        "scalar_z=ax+by+c",
+        # array data function names
+        "array_y=x+k",
+        "array_y=mx",
+        "array_y=mx+c",
+        "array_z=ax+by+c"
+    ]
+
+
+@pytest.fixture
+def function_arities():
+    return [
+        1,
+        2,
+        3,
+        5,
+    ]
+
+
+@pytest.fixture
+def function_docs():
+    return [
+        unary_doc,
+        binary_doc,
+        ternary_doc,
+        varargs_doc
+    ]
+
+
+@pytest.fixture
+def functions():
+    return [
+        unary_function,
+        binary_function,
+        ternary_function,
+        varargs_function
+    ]
+
+
+@pytest.fixture
+def function_inputs():
+    return [
+        # scalar input data
+        [
+            pa.scalar(10, pa.int64())
+        ],
+        [
+            pa.scalar(10, pa.int64()),
+            pa.scalar(2, pa.int64())
+        ],
+        [
+            pa.scalar(10, pa.int64()),
+            pa.scalar(2, pa.int64()),
+            pa.scalar(5, pa.int64())
+        ],
+        [
+            pa.scalar(2, pa.int64()),
+            pa.scalar(10, pa.int64()),
+            pa.scalar(3, pa.int64()),
+            pa.scalar(20, pa.int64()),
+            pa.scalar(5, pa.int64())
+        ],
+        # array input data
+        [
+            pa.array([10, 20], pa.int64())
+        ],
+        [
+            pa.array([10, 20], pa.int64()),
+            pa.array([2, 4], pa.int64())
+        ],
+        [
+            pa.array([10, 20], pa.int64()),
+            pa.array([2, 4], pa.int64()),
+            pa.array([5, 10], pa.int64())
+        ],
+        [
+            pa.array([2, 3], pa.int64()),
+            pa.array([10, 20], pa.int64()),
+            pa.array([3, 7], pa.int64()),
+            pa.array([20, 30], pa.int64()),
+            pa.array([5, 10], pa.int64())
+        ]
+    ]
+
+
+@pytest.fixture
+def expected_outputs():
+    return [
+        # scalar output data
+        pa.scalar(11, pa.int64()),  # 10 + 1
+        pa.scalar(20, pa.int64()),  # 10 * 2
+        pa.scalar(25, pa.int64()),  # 10 * 2 + 5
+        pa.scalar(85, pa.int64()),  # (2 * 10) + (3 * 20) + 5
+        # array output data
+        pa.array([11, 21], pa.int64()),  # [10 + 1, 20 + 1]
+        pa.array([20, 80], pa.int64()),  # [10 * 2, 20 * 4]
+        pa.array([25, 90], pa.int64()),  # [(10 * 2) + 5, (20 * 4) + 10]
+        # [(2 * 10) + (3 * 20) + 5, (3 * 20) + (7 * 30) + 10]
+        pa.array([85, 280], pa.int64())
+    ]
+
+
+def test_scalar_udf_function_with_scalar_data(function_names,

Review comment:
       I was trying to reproduce an error on this test and I noticed that I can't repeat this test because it modifies the global state (adds a function to the global function registry) and so the second run fails (function already exists).
   
   I don't know a great way to fix this.  We could expose the ability to create new function registries to python but that might be a bit of work and we would need to update the register function to take a new registry argument that is rarely specified so it adds complexity.
   
   We could also add the ability to unregister a function.  That might be the easiest.  Since function names have to be unique it should be straightforward to unregister a function by name.  I don't know if either of those things have to be fixed as part of this PR but I was curious to get your thoughts.

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }

Review comment:
       Nit: These checks could probably be the first things we do (before the `Py_INCREF`)

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);

Review comment:
       Can we / should we confirm that `result` is a scalar here?

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }
+  if (!PyCallable_Check(function_.obj())) {
+    return Status::TypeError("Expected a callable python object.");
+  }
+  auto doc = options->doc();
+  auto arity = options->arity();
+  scalar_func_ = std::make_shared<compute::ScalarFunction>(options->name(), arity, &doc);
+
+  // lambda function
+  auto call_back = [&, arity](compute::KernelContext* ctx,

Review comment:
       The default capture specifier of `&` is a little misleading.  The only thing you are capturing with this flag is `this` and `this` is never captured by reference.  When I see a default capture of `&` it usually tells me that the lambda is going to only be executed within the current scope (otherwise anything it's referencing would be out of scope) and I know this lambda is going to be executed outside this scope so it is strange to see.
   
   I often like to be specific when I'm not capturing too many things.  You could use `[this, arity]` to be more clear.
   

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }
+  if (!PyCallable_Check(function_.obj())) {
+    return Status::TypeError("Expected a callable python object.");
+  }
+  auto doc = options->doc();
+  auto arity = options->arity();
+  scalar_func_ = std::make_shared<compute::ScalarFunction>(options->name(), arity, &doc);

Review comment:
       doc is a stack allocated variable that will be destroyed when this function exits.  It is not safe to take the address to it in this way.  More on this in the top-level review comment.  This is most likely what is causing the python tests to fail.

##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace arrow {
+
+namespace py {
+
+// Exposing the UDFOptions: https://issues.apache.org/jira/browse/ARROW-16041
+class ARROW_PYTHON_EXPORT UdfOptions {
+ public:
+  UdfOptions(const compute::Function::Kind kind, const compute::Arity arity,
+             const compute::FunctionDoc func_doc,
+             const std::vector<compute::InputType> in_types,
+             const compute::OutputType out_type,
+             const compute::MemAllocation::type mem_allocation,
+             const compute::NullHandling::type null_handling)
+      : kind_(kind),
+        arity_(arity),
+        func_doc_(func_doc),
+        in_types_(in_types),
+        out_type_(out_type),
+        mem_allocation_(mem_allocation),
+        null_handling_(null_handling) {}
+
+  compute::Function::Kind kind() { return kind_; }
+
+  const compute::Arity& arity() const { return arity_; }
+
+  const compute::FunctionDoc doc() const { return func_doc_; }
+
+  const std::vector<compute::InputType>& input_types() const { return in_types_; }
+
+  const compute::OutputType& output_type() const { return out_type_; }
+
+  compute::MemAllocation::type mem_allocation() { return mem_allocation_; }
+
+  compute::NullHandling::type null_handling() { return null_handling_; }

Review comment:
       Similiarly, we should hardcode this to `COMPUTED_NO_PREALLOCATE` and hide this detail from the user.  I don't think the other options make sense if we are doing `NO_PREALLOCATE`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831742675



##########
File path: python/pyarrow/tests/test_udf.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import List
+
+import pytest
+
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+# scalar unary function data
+
+
+unary_doc = get_function_doc("add function",
+                             "test add function",
+                             ["scalar1"],
+                             "None")
+
+
+def unary_function(scalar1):

Review comment:
       Yes, that's right. I will add cases for failures. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831739519



##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,322 @@
+from typing import List
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+
+"""
+Array Usage
+"""
+
+# Example 1: Array Unary
+print("=" * 80)
+print("Example 1: Array Unary")
+print("=" * 80)
+
+
+def add_constant(array):
+    return pc.call_function("add", [array, 1])
+
+
+func_name_1 = "py_add_func"
+arity_1 = Arity.unary()
+in_types_1 = [InputType.array(pa.int64())]
+out_type_1 = pa.int64()
+doc_1 = get_function_doc("add function", "test add function",
+                         ["value"], "None")

Review comment:
       of course. that's better. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r836096042



##########
File path: python/pyarrow/_compute.pyx
##########
@@ -479,6 +611,9 @@ cdef class FunctionRegistry(_Weakrefable):
             func = GetResultValue(self.registry.GetFunction(c_name))
         return wrap_function(func)
 
+    def register_function(self, name, arity, input_types, output_type, function_kind):
+        pass
+

Review comment:
       Typo. Removing it. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r837631893



##########
File path: cpp/src/arrow/python/api.h
##########
@@ -28,3 +28,4 @@
 #include "arrow/python/numpy_to_arrow.h"
 #include "arrow/python/python_to_arrow.h"
 #include "arrow/python/serialize.h"
+#include "arrow/python/udf.h"

Review comment:
       Yes, this would be only used in Cython. You’re correct. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r836422695



##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DECLARE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)               \
+  ARROW_PYTHON_EXPORT Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch&, \
+                                                             PyObject*, int, Datum*);
+
+DECLARE_CALL_UDF(Scalar, scalar, scalar)
+DECLARE_CALL_UDF(Array, array, make_array)
+
+#undef DECLARE_CALL_UDF
+
+class ARROW_PYTHON_EXPORT UdfBuilder {

Review comment:
       For this PR I added a struct, but need to evaluate how this needs to be exposed to Python. Added a separate JIRA to work on it: https://issues.apache.org/jira/browse/ARROW-16041




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r837636790



##########
File path: python/pyarrow/public-api.pxi
##########
@@ -38,7 +37,6 @@ cdef api shared_ptr[CBuffer] pyarrow_unwrap_buffer(object buffer):
 
     return shared_ptr[CBuffer]()
 
-

Review comment:
       Formatting issue. This is a typo.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839186689



##########
File path: cpp/examples/arrow/CMakeLists.txt
##########
@@ -133,4 +133,11 @@ if(ARROW_PARQUET AND ARROW_DATASET)
 
   add_arrow_example(join_example EXTRA_LINK_LIBS ${DATASET_EXAMPLES_LINK_LIBS})
   add_dependencies(join-example parquet)
+
+  add_arrow_example(udf_example EXTRA_LINK_LIBS ${DATASET_EXAMPLES_LINK_LIBS})
+  add_dependencies(udf-example parquet)
+
+  add_arrow_example(aggregate_example EXTRA_LINK_LIBS ${DATASET_EXAMPLES_LINK_LIBS})
+  add_dependencies(aggregate-example parquet)

Review comment:
       For udf one it is not needed, you're correct. I also removed the aggregate example from this PR. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1084197255


   @vibhatha I'm not entirely clear on the concern.  I don't mind if we have an inheritence tree of object structs.  For example, this would be ok (this is totally made up and likely is not correct for aggregate UDFs at all):
   
   ```
   namespace arrow {
   namespace py {
   namespace compute {
   
   struct UdfOptions {
     cp::Function::Kind kind;
     cp::Arity arity;
     const cp::FunctionDoc func_doc;
     std::vector<cp::InputType> in_types;
     cp::OutputType out_type;
     cp::MemAllocation::type mem_allocation;
     cp::NullHandling::type null_handling;
   };
   
   struct ScalarUdfOptions : public UdfOptions {
     std::string func_name;
   };
   
   struct AggregateUdfOptions : public UdfOptions {
     std::string init_func_name;
     std::string map_func_name;
     std::string reduce_func_name;
   };
   
   Status MakeScalarFunction(PyObject* function, const ScalarUdfOptions& options);
   Status MakeAggregateFunction(PyObject* function, const AggregateUdfOptions& options);
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840995501



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_scalar(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status exec_function_array(const compute::ExecBatch& batch, PyObject* function,
+                           int num_args, Datum* out) {
+  std::shared_ptr<Array> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_array()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].make_array();
+    PyObject* data = wrap_array(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);
+  if (result == NULL) {
+    return Status::ExecutionError("Error occured in computation");
+  }
+  auto res = unwrap_array(result);
+  if (!res.status().ok()) {
+    return res.status();
+  }
+  c_res_data = res.ValueOrDie();
+  auto datum = new Datum(c_res_data);
+  *out = *datum;
+  return Status::OK();
+}
+
+Status VerifyArityAndInput(compute::Arity arity, const compute::ExecBatch& batch) {
+  bool match = static_cast<uint64_t>(arity.num_args) == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, ScalarUdfOptions* options) {
+  // creating a copy of objects for the lambda function
+  Py_INCREF(function);
+  function_.reset(function);
+  if (function_.obj() == NULL) {
+    return Status::ExecutionError("python function cannot be null");
+  }
+  if (!PyCallable_Check(function_.obj())) {
+    return Status::TypeError("Expected a callable python object.");
+  }
+  auto doc = options->doc();
+  auto arity = options->arity();
+  scalar_func_ = std::make_shared<compute::ScalarFunction>(options->name(), arity, &doc);
+
+  // lambda function
+  auto call_back = [&, arity](compute::KernelContext* ctx,

Review comment:
       Yes that sounds better and acutally this is not a callback. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r840995546



##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace arrow {
+
+namespace py {
+
+// Exposing the UDFOptions: https://issues.apache.org/jira/browse/ARROW-16041
+class ARROW_PYTHON_EXPORT UdfOptions {
+ public:
+  UdfOptions(const compute::Function::Kind kind, const compute::Arity arity,
+             const compute::FunctionDoc func_doc,
+             const std::vector<compute::InputType> in_types,
+             const compute::OutputType out_type,
+             const compute::MemAllocation::type mem_allocation,
+             const compute::NullHandling::type null_handling)
+      : kind_(kind),
+        arity_(arity),
+        func_doc_(func_doc),
+        in_types_(in_types),
+        out_type_(out_type),
+        mem_allocation_(mem_allocation),
+        null_handling_(null_handling) {}
+
+  compute::Function::Kind kind() { return kind_; }
+
+  const compute::Arity& arity() const { return arity_; }
+
+  const compute::FunctionDoc doc() const { return func_doc_; }
+
+  const std::vector<compute::InputType>& input_types() const { return in_types_; }
+
+  const compute::OutputType& output_type() const { return out_type_; }
+
+  compute::MemAllocation::type mem_allocation() { return mem_allocation_; }

Review comment:
       Sure, we can do that. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r841146632



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace arrow {
+
+namespace py {
+
+Status exec_function_scalar(const compute::ExecBatch& batch, PyObject* function,
+                            int num_args, Datum* out) {
+  std::shared_ptr<Scalar> c_res_data;
+  PyObject* arg_tuple = PyTuple_New(num_args);
+  for (int arg_id = 0; arg_id < num_args; arg_id++) {
+    if (!batch[arg_id].is_scalar()) {
+      return Status::Invalid("Input type and data type doesn't match");
+    }
+    auto c_data = batch[arg_id].scalar();
+    PyObject* data = wrap_scalar(c_data);
+    PyTuple_SetItem(arg_tuple, arg_id, data);
+  }
+  PyObject* result = PyObject_CallObject(function, arg_tuple);

Review comment:
       Need to check if we can unwrap the `PyObject*` to a `Datum` and evaluate whether is it a scalar or not. Is there a better way? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839546515



##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expected {}");
+  }
+  return Status::OK();
+}
+
+Status ScalarUdfBuilder::MakeFunction(PyObject* function, UDFOptions* options) {
+  Status st;
+  auto doc = this->doc();
+  auto func = std::make_shared<cp::ScalarFunction>(this->name(), this->arity(), &doc);
+  // creating a copy of objects for the lambda function
+  auto py_function = function;

Review comment:
       I used the `OwnedRefNoGIL` could you please check if it is used the intended way :) 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#issuecomment-1086326440


   I would probably prefer (C), I would guess that the reason why FunctionDoc is stored in Function as a pointer and not a smart pointer is just that it made it easier to declare the documentation as a static


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r839484191



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions&, const cp::FunctionOptions&) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(const cp::FunctionOptions&) const override;

Review comment:
       Removed. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r831714645



##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This example showcases various ways to work with Datasets. It's
+// intended to be paired with the documentation.

Review comment:
       ```suggestion
   ```

##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DECLARE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)               \
+  ARROW_PYTHON_EXPORT Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch&, \
+                                                             PyObject*, int, Datum*);
+
+DECLARE_CALL_UDF(Scalar, scalar, scalar)
+DECLARE_CALL_UDF(Array, array, make_array)

Review comment:
       This macro isn't saving us much at all.  Can we do away with it?

##########
File path: cpp/src/arrow/python/udf.h
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/python/platform.h"
+
+#include <cstdint>
+#include <memory>
+
+#include "arrow/compute/api_scalar.h"
+#include "arrow/compute/cast.h"
+#include "arrow/compute/exec.h"
+#include "arrow/compute/function.h"
+#include "arrow/compute/registry.h"
+#include "arrow/datum.h"
+#include "arrow/util/cpu_info.h"
+#include "arrow/util/logging.h"
+
+#include "arrow/python/common.h"
+#include "arrow/python/pyarrow.h"
+#include "arrow/python/visibility.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DECLARE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)               \
+  ARROW_PYTHON_EXPORT Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch&, \
+                                                             PyObject*, int, Datum*);
+
+DECLARE_CALL_UDF(Scalar, scalar, scalar)
+DECLARE_CALL_UDF(Array, array, make_array)
+
+#undef DECLARE_CALL_UDF
+
+class ARROW_PYTHON_EXPORT UdfBuilder {

Review comment:
       I think I'd rather see an options object (`UdfOptions`) and a function `Status RegisterScalarUdf(PyObject* function, const UdfOptions& udf_options);`
   
   The options object could then be a plain struct with public fields.
   

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \

Review comment:
       This is clever but we generally try to avoid macros when possible (this is to allow you to write the function once instead of two very similar versions right?) as they make reading the code a bit more difficult.  I'm a little on the fence regarding this macro because we only have two invocations.
   
   CC @pitrou for a second opinion

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)
+        )
+    in_types: List[InputType]
+        list of InputType objects which defines the input 
+        types for the function
+    out_type: DataType
+        output type of the function
+    callback: callable
+        user defined function 
+    mem_allocation: str
+        memory allocation mode 
+        "preallocate" or "no_preallocate"

Review comment:
       What does this mean?

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)
+        )
+    in_types: List[InputType]
+        list of InputType objects which defines the input 
+        types for the function
+    out_type: DataType
+        output type of the function
+    callback: callable
+        user defined function 
+    mem_allocation: str
+        memory allocation mode 
+        "preallocate" or "no_preallocate"
+    null_handling: str
+        null handling mode
+        one of "intersect", "computed_preallocate",
+        "computed_no_preallocate", 
+        "output_not_null"

Review comment:
       What does this mean?

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow

Review comment:
       ```suggestion
   // Demonstrate registering a user-defined Arrow compute function outside of the Arrow
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>

Review comment:
       Same comment as before.  It seems we have too many `#include`

##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This example showcases various ways to work with Datasets. It's
+// intended to be paired with the documentation.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+

Review comment:
       We aren't stellar at IWYU but we should try and maintain it for our example code.  I think we also need...
   
   ```
   #include <type_traits> // for std::enable_if
   #include <vector> // for std::vector
   ```

##########
File path: cpp/src/arrow/python/udf.cc
##########
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/python/udf.h"
+
+#include <cstddef>
+#include <memory>
+#include <sstream>
+
+#include "arrow/compute/function.h"
+#include "arrow/python/common.h"
+
+namespace cp = arrow::compute;
+
+namespace arrow {
+
+namespace py {
+
+#define DEFINE_CALL_UDF(TYPE_NAME, FUNCTION_SUFFIX, CONVERT_SUFFIX)                      \
+  Status exec_function_##FUNCTION_SUFFIX(const cp::ExecBatch& batch, PyObject* function, \
+                                         int num_args, Datum* out) {                     \
+    std::shared_ptr<TYPE_NAME> c_res_data;                                               \
+    PyObject* arg_tuple = PyTuple_New(num_args);                                         \
+    for (int arg_id = 0; arg_id < num_args; arg_id++) {                                  \
+      if (!batch[arg_id].is_##FUNCTION_SUFFIX()) {                                       \
+        return Status::Invalid("Input type and data type doesn't match");                \
+      }                                                                                  \
+      auto c_data = batch[arg_id].CONVERT_SUFFIX();                                      \
+      PyObject* data = wrap_##FUNCTION_SUFFIX(c_data);                                   \
+      PyTuple_SetItem(arg_tuple, arg_id, data);                                          \
+    }                                                                                    \
+    PyObject* result = PyObject_CallObject(function, arg_tuple);                         \
+    if (result == NULL) {                                                                \
+      return Status::ExecutionError("Error occured in computation");                     \
+    }                                                                                    \
+    auto res = unwrap_##FUNCTION_SUFFIX(result);                                         \
+    if (!res.status().ok()) {                                                            \
+      return res.status();                                                               \
+    }                                                                                    \
+    c_res_data = res.ValueOrDie();                                                       \
+    auto datum = new Datum(c_res_data);                                                  \
+    *out = *datum;                                                                       \
+    return Status::OK();                                                                 \
+  }
+
+DEFINE_CALL_UDF(Scalar, scalar, scalar)
+DEFINE_CALL_UDF(Array, array, make_array)
+
+#undef DEFINE_CALL_UDF
+
+Status VerifyArityAndInput(cp::Arity arity, const cp::ExecBatch& batch) {
+  bool match = (uint64_t)arity.num_args == batch.values.size();
+  if (!match) {
+    return Status::Invalid(
+        "Function Arity and Input data shape doesn't match, expceted {}");

Review comment:
       ```suggestion
           "Function Arity and Input data shape doesn't match, expected {}");
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions&, const cp::FunctionOptions&) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(const cp::FunctionOptions&) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {
+ public:
+  ExampleNode(ExecNode* input, const ExampleNodeOptions&)
+      : ExecNode(/*plan=*/input->plan(), /*inputs=*/{input},
+                 /*input_labels=*/{"ignored"},
+                 /*output_schema=*/input->output_schema(), /*num_outputs=*/1) {}
+
+  const char* kind_name() const override { return "ExampleNode"; }
+
+  arrow::Status StartProducing() override {
+    outputs_[0]->InputFinished(this, 0);
+    return arrow::Status::OK();
+  }
+
+  void ResumeProducing(ExecNode* output) override {}
+  void PauseProducing(ExecNode* output) override {}
+
+  void StopProducing(ExecNode* output) override { inputs_[0]->StopProducing(this); }
+  void StopProducing() override { inputs_[0]->StopProducing(); }
+
+  void InputReceived(ExecNode* input, cp::ExecBatch batch) override {}
+  void ErrorReceived(ExecNode* input, arrow::Status error) override {}
+  void InputFinished(ExecNode* input, int total_batches) override {}
+
+  arrow::Future<> finished() override { return inputs_[0]->finished(); }
+};
+
+arrow::Result<cp::ExecNode*> ExampleExecNodeFactory(cp::ExecPlan* plan,
+                                                    std::vector<cp::ExecNode*> inputs,
+                                                    const cp::ExecNodeOptions& options) {
+  const auto& example_options =
+      arrow::internal::checked_cast<const ExampleNodeOptions&>(options);
+
+  return plan->EmplaceNode<ExampleNode>(inputs[0], example_options);
+}
+
+const cp::FunctionDoc func_doc{
+    "User-defined-function usage to demonstrate registering an out-of-tree function",
+    "returns x + y + z",
+    {"x", "y", "z"},
+    "UDFOptions"};
+
+arrow::Status Execute() {
+  const std::string name = "x+x";
+  auto func = std::make_shared<cp::ScalarFunction>(name, cp::Arity::Ternary(), &func_doc);
+
+  auto exec_func = [](cp::KernelContext* ctx, const cp::ExecBatch& batch,

Review comment:
       Can you make this an actual function instead of a lambda?

##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This example showcases various ways to work with Datasets. It's
+// intended to be paired with the documentation.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+namespace cp = arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {arrow::field("a", arrow::int64()),
+                       arrow::field("b", arrow::boolean()),
+                       arrow::field("c", arrow::int64())};
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({0, 1, 2, 0, 4, 1, 0, 5}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array,
+                        GetArrayDataSample<arrow::BooleanType>(
+                            {false, true, false, true, true, false, true, false}));
+  ARROW_ASSIGN_OR_RAISE(auto data_array, GetArrayDataSample<arrow::Int64Type>(
+                                             {10, 11, 12, 10, 11, 11, 10, 15}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, bool_array, data_array};
+
+  table = arrow::Table::Make(schema, data_vector, 8);
+
+  return table;
+}
+
+arrow::Status DoAggregate() {
+  auto maybe_plan = cp::ExecPlan::Make();
+  ABORT_ON_FAILURE(maybe_plan.status());
+  auto plan = maybe_plan.ValueOrDie();
+  cp::ExecContext exec_context(arrow::default_memory_pool(),
+                               ::arrow::internal::GetCpuThreadPool());
+
+  ARROW_ASSIGN_OR_RAISE(auto table, GetTable());
+
+  std::cout << "Source Table" << std::endl;
+
+  std::cout << table->ToString() << std::endl;
+
+  std::shared_ptr<arrow::Table> out;
+  cp::CountOptions options(cp::CountOptions::ONLY_VALID);
+  auto aggregate_options = cp::AggregateNodeOptions{/*aggregates=*/{{"sum", &options}},
+                                                    /*targets=*/{"c"},
+                                                    /*names=*/{"count(c)"},
+                                                    /*keys=*/{}};
+  auto schema = arrow::schema({arrow::field("count(c)", arrow::int64())});
+
+  ABORT_ON_FAILURE(cp::Declaration::Sequence(
+                       {
+                           {"table_source", cp::TableSourceNodeOptions{table, 2}},
+                           {"aggregate", aggregate_options},
+                           {"table_sink", cp::TableSinkNodeOptions{&out, schema}},
+                       })
+                       .AddToPlan(plan.get())
+                       .status());
+
+  ARROW_RETURN_NOT_OK(plan->StartProducing());
+
+  std::cout << "Output Table Data : " << std::endl;
+  std::cout << out->ToString() << std::endl;
+
+  auto future = plan->finished();
+
+  return future.status();
+}
+
+int main(int argc, char** argv) {

Review comment:
       How is this example different than `SourceScalarAggregateSinkExample`?  I don't know that it needs to be but I'm wondering the motivation here.

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -199,6 +202,135 @@ FunctionDoc = namedtuple(
      "options_required"))
 
 
+cdef wrap_arity(const CArity c_arity):
+    """
+    Wrap a C++ Arity in an Arity object
+    """
+    cdef Arity arity = Arity.__new__(Arity)
+    arity.init(c_arity)
+    return arity
+
+
+cdef wrap_input_type(const CInputType c_input_type):
+    """
+    Wrap a C++ InputType in an InputType object
+    """
+    cdef InputType input_type = InputType.__new__(InputType)
+    input_type.init(c_input_type)
+    return input_type
+
+
+cdef class InputType(_Weakrefable):
+    """
+    An interface for defining input-types for streaming execution engine
+    applications. 
+    """
+
+    def __init__(self):
+        raise TypeError("Cannot use constructor to initialize InputType")
+
+    cdef void init(self, const CInputType &input_type):
+        self.input_type = input_type
+
+    @staticmethod
+    def scalar(data_type):
+        """
+        create a scalar input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.scalar(pa.int32())
+        <pyarrow._compute.InputType object at 0x1029fdcb0>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Scalar(c_data_type)
+        return wrap_input_type(c_input_type)
+
+    @staticmethod
+    def array(data_type):
+        """
+        create an array input type of the given data type
+
+        Parameter
+        ---------
+        data_type: DataType
+
+        Examples
+        --------
+
+        >>> import pyarrow as pa
+        >>> from pyarrow.compute import InputType
+        >>> in_type = InputType.array(pa.int32())
+        <pyarrow._compute.InputType object at 0x102ba4850>
+        """
+        cdef:
+            shared_ptr[CDataType] c_data_type
+            CInputType c_input_type
+        c_data_type = pyarrow_unwrap_data_type(data_type)
+        c_input_type = CInputType.Array(c_data_type)
+        return wrap_input_type(c_input_type)
+
+
+cdef class Arity(_Weakrefable):

Review comment:
       Why can't arity just be an integer?

##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,322 @@
+from typing import List
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+
+"""
+Array Usage
+"""
+
+# Example 1: Array Unary
+print("=" * 80)
+print("Example 1: Array Unary")
+print("=" * 80)
+
+
+def add_constant(array):
+    return pc.call_function("add", [array, 1])
+
+
+func_name_1 = "py_add_func"
+arity_1 = Arity.unary()
+in_types_1 = [InputType.array(pa.int64())]
+out_type_1 = pa.int64()
+doc_1 = get_function_doc("add function", "test add function",
+                         ["value"], "None")

Review comment:
       ```suggestion
   doc_1 = get_function_doc("add one function", "adds one to values",
                            ["value"], "None")
   ```
   Let's pretend this is a real case instead of "test add function" to make it more clear the intent of the field.

##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review comment:
       Since this PR is for scalar UDF could we add this example in a separate PR?

##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,322 @@
+from typing import List
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+
+"""
+Array Usage
+"""
+
+# Example 1: Array Unary
+print("=" * 80)
+print("Example 1: Array Unary")
+print("=" * 80)
+
+
+def add_constant(array):
+    return pc.call_function("add", [array, 1])

Review comment:
       ```suggestion
   def add_one(array):
       return pc.call_function("add", [array, 1])
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions&, const cp::FunctionOptions&) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(const cp::FunctionOptions&) const override;
+};
+
+cp::FunctionOptionsType* GetUDFOptionsType() {
+  static UDFOptionsType options_type;
+  return &options_type;
+}
+
+class UDFOptions : public cp::FunctionOptions {
+ public:
+  UDFOptions() : cp::FunctionOptions(GetUDFOptionsType()) {}
+};
+
+std::unique_ptr<cp::FunctionOptions> UDFOptionsType::Copy(
+    const cp::FunctionOptions&) const {
+  return std::unique_ptr<cp::FunctionOptions>(new UDFOptions());
+}
+
+class ExampleNodeOptions : public cp::ExecNodeOptions {};
+
+// a basic ExecNode which ignores all input batches
+class ExampleNode : public cp::ExecNode {
+ public:
+  ExampleNode(ExecNode* input, const ExampleNodeOptions&)
+      : ExecNode(/*plan=*/input->plan(), /*inputs=*/{input},
+                 /*input_labels=*/{"ignored"},
+                 /*output_schema=*/input->output_schema(), /*num_outputs=*/1) {}
+
+  const char* kind_name() const override { return "ExampleNode"; }
+
+  arrow::Status StartProducing() override {
+    outputs_[0]->InputFinished(this, 0);
+    return arrow::Status::OK();
+  }
+
+  void ResumeProducing(ExecNode* output) override {}
+  void PauseProducing(ExecNode* output) override {}
+
+  void StopProducing(ExecNode* output) override { inputs_[0]->StopProducing(this); }
+  void StopProducing() override { inputs_[0]->StopProducing(); }
+
+  void InputReceived(ExecNode* input, cp::ExecBatch batch) override {}
+  void ErrorReceived(ExecNode* input, arrow::Status error) override {}
+  void InputFinished(ExecNode* input, int total_batches) override {}
+
+  arrow::Future<> finished() override { return inputs_[0]->finished(); }
+};
+
+arrow::Result<cp::ExecNode*> ExampleExecNodeFactory(cp::ExecPlan* plan,
+                                                    std::vector<cp::ExecNode*> inputs,
+                                                    const cp::ExecNodeOptions& options) {
+  const auto& example_options =
+      arrow::internal::checked_cast<const ExampleNodeOptions&>(options);
+
+  return plan->EmplaceNode<ExampleNode>(inputs[0], example_options);
+}
+
+const cp::FunctionDoc func_doc{
+    "User-defined-function usage to demonstrate registering an out-of-tree function",
+    "returns x + y + z",
+    {"x", "y", "z"},
+    "UDFOptions"};
+
+arrow::Status Execute() {
+  const std::string name = "x+x";

Review comment:
       Why `x+x` and not `x+y+z`  Or maybe `add_three` or something so it's obvious the difference between the projected field name (the key in your expression dictionary passed into project) and the function name (what is passed into `call`).

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -479,6 +611,9 @@ cdef class FunctionRegistry(_Weakrefable):
             func = GetResultValue(self.registry.GetFunction(c_name))
         return wrap_function(func)
 
+    def register_function(self, name, arity, input_types, output_type, function_kind):
+        pass
+

Review comment:
       What's this?

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 

Review comment:
       ```suggestion
       Register a user-defined-function 
   ```
   What's `(function)`?

##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,322 @@
+from typing import List
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+
+"""
+Array Usage
+"""
+
+# Example 1: Array Unary
+print("=" * 80)
+print("Example 1: Array Unary")
+print("=" * 80)
+
+
+def add_constant(array):
+    return pc.call_function("add", [array, 1])
+
+
+func_name_1 = "py_add_func"

Review comment:
       ```suggestion
   func_name_1 = "py_add_one_func"
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}

Review comment:
       Is this used?

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 

Review comment:
       Does this need to be unique?  Can I register multiple functions with different arity / in_types but the same name?

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions&, const cp::FunctionOptions&) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(const cp::FunctionOptions&) const override;

Review comment:
       Where's the implementation?

##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,322 @@
+from typing import List
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+
+"""
+Array Usage
+"""
+
+# Example 1: Array Unary
+print("=" * 80)
+print("Example 1: Array Unary")
+print("=" * 80)
+
+
+def add_constant(array):
+    return pc.call_function("add", [array, 1])
+
+
+func_name_1 = "py_add_func"
+arity_1 = Arity.unary()

Review comment:
       Can we use scopes or something to get rid of all these variable suffixes?  E.g.
   
   ```
   def array_unary_example():
     func_name = ...
     arity = ...
   ```

##########
File path: python/pyarrow/public-api.pxi
##########
@@ -25,11 +25,9 @@ from pyarrow.includes.libarrow cimport (CArray, CDataType, CField,
 # You cannot assign something to a dereferenced pointer in Cython thus these
 # methods don't use Status to indicate a successful operation.
 
-

Review comment:
       Let's avoid formatting changes to unrelated files

##########
File path: python/pyarrow/tests/test_udf.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import List
+
+import pytest
+
+import pyarrow as pa
+from pyarrow import compute as pc
+from pyarrow.compute import register_function
+from pyarrow.compute import Arity, InputType
+
+
+def get_function_doc(summary: str, desc: str, arg_names: List[str],
+                     options_class: str, options_required: bool = False):
+    func_doc = {}
+    func_doc["summary"] = summary
+    func_doc["description"] = desc
+    func_doc["arg_names"] = arg_names
+    func_doc["options_class"] = options_class
+    func_doc["options_required"] = False
+    return func_doc
+
+# scalar unary function data
+
+
+unary_doc = get_function_doc("add function",
+                             "test add function",
+                             ["scalar1"],
+                             "None")
+
+
+def unary_function(scalar1):

Review comment:
       We should eventually add tests for negative cases as well (missing arguments, invalid values, etc.)

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)
+        )
+    in_types: List[InputType]
+        list of InputType objects which defines the input 
+        types for the function
+    out_type: DataType
+        output type of the function
+    callback: callable
+        user defined function 

Review comment:
       What does this function get passed in?  What does it need to return?  We can find this information out from the examples but this seems a little sparse.

##########
File path: cpp/examples/arrow/aggregate_example.cc
##########
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This example showcases various ways to work with Datasets. It's
+// intended to be paired with the documentation.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>

Review comment:
       Seems odd that I would still need to `#include` this many files.  I think it should be sufficient to do:
   ```
   #include <arrow/api.h>
   #include <arrow/compute/api.h>
   #include <arrow/compute/exec/exec_plan.h> // ARROW-15263
   ```

##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+

Review comment:
       ```
   #include <vector> // std::vector
   #include <string> // std::string
   #include <utility> // std::move
   ```

##########
File path: python/pyarrow/_compute.pyx
##########
@@ -2179,3 +2314,205 @@ cdef CExpression _bind(Expression filter, Schema schema) except *:
 
     return GetResultValue(filter.unwrap().Bind(
         deref(pyarrow_unwrap_schema(schema).get())))
+
+
+cdef CFunctionDoc _make_function_doc(func_doc):
+    """
+    Helper function to generate the FunctionDoc
+    """
+    cdef:
+        CFunctionDoc f_doc
+        vector[c_string] c_arg_names
+        c_bool c_options_required
+    if isinstance(func_doc, dict):
+        if func_doc["summary"] and isinstance(func_doc["summary"], str):
+            f_doc.summary = func_doc["summary"].encode()
+        else:
+            raise ValueError("key `summary` cannot be None")
+
+        if func_doc["description"] and isinstance(func_doc["description"], str):
+            f_doc.description = func_doc["description"].encode()
+        else:
+            raise ValueError("key `description` cannot be None")
+
+        if func_doc["arg_names"] and isinstance(func_doc["arg_names"], list):
+            for arg_name in func_doc["arg_names"]:
+                if isinstance(arg_name, str):
+                    c_arg_names.push_back(arg_name.encode())
+                else:
+                    raise ValueError(
+                        "key `arg_names` must be a list of strings")
+            f_doc.arg_names = c_arg_names
+        else:
+            raise ValueError("key `arg_names` cannot be None")
+
+        if func_doc["options_class"] and isinstance(func_doc["options_class"], str):
+            f_doc.options_class = func_doc["options_class"].encode()
+        else:
+            raise ValueError("key `options_class` cannot be None")
+
+        if isinstance(func_doc["options_required"], bool):
+            c_options_required = func_doc["options_required"]
+            f_doc.options_required = c_options_required
+        else:
+            raise ValueError("key `options_required` must be bool")
+
+        return f_doc
+    else:
+        raise TypeError(f"func_doc must be a dictionary")
+
+
+cdef class UDFError(Exception):
+    cdef dict __dict__
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message)
+        self.extra_info = tobytes(extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+cdef class UDFRegistrationError(UDFError):
+
+    def __init__(self, message='', extra_info=b''):
+        super().__init__(message, extra_info)
+
+    cdef CStatus to_status(self):
+        message = tobytes("UDF Registration error: {}".format(str(self)))
+        return CStatus_UnknownError(message)
+
+
+def register_function(func_name, arity, function_doc, in_types,
+                      out_type, callback, mem_allocation="no_preallocate",
+                      null_handling="computed_no_preallocate"):
+    """
+    Register a user-defined-function (function) 
+
+    Parameters
+    ----------
+
+    func_name: str
+        function name 
+    arity: Arity
+        arity of the function
+    function_doc: dict
+        a dictionary object with keys 
+        ("summary", 
+        "description", 
+        "arg_names", 
+        "options_class", (not supported yet)
+        "options_required" (not supported yet)

Review comment:
       It's ok to do this in pieces but can we just leave these keys out instead of saying "not supported yet"?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r836068647



##########
File path: cpp/examples/arrow/udf_example.cc
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <arrow/compute/api.h>
+#include <arrow/compute/exec/exec_plan.h>
+#include <arrow/compute/exec/expression.h>
+#include <arrow/compute/exec/options.h>
+#include <arrow/datum.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/status.h>
+#include <arrow/table.h>
+#include <arrow/util/async_generator.h>
+#include <arrow/util/future.h>
+#include <arrow/util/vector.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <memory>
+
+// Demonstrate registering an user-defined Arrow compute function outside of the Arrow
+// source tree
+
+namespace cp = ::arrow::compute;
+
+#define ABORT_ON_FAILURE(expr)                     \
+  do {                                             \
+    arrow::Status status_ = (expr);                \
+    if (!status_.ok()) {                           \
+      std::cerr << status_.message() << std::endl; \
+      abort();                                     \
+    }                                              \
+  } while (0);
+
+template <typename TYPE,
+          typename = typename std::enable_if<arrow::is_number_type<TYPE>::value |
+                                             arrow::is_boolean_type<TYPE>::value |
+                                             arrow::is_temporal_type<TYPE>::value>::type>
+arrow::Result<std::shared_ptr<arrow::Array>> GetArrayDataSample(
+    const std::vector<typename TYPE::c_type>& values) {
+  using ARROW_ARRAY_TYPE = typename arrow::TypeTraits<TYPE>::ArrayType;
+  using ARROW_BUILDER_TYPE = typename arrow::TypeTraits<TYPE>::BuilderType;
+  ARROW_BUILDER_TYPE builder;
+  ARROW_RETURN_NOT_OK(builder.Reserve(values.size()));
+  std::shared_ptr<ARROW_ARRAY_TYPE> array;
+  ARROW_RETURN_NOT_OK(builder.AppendValues(values));
+  ARROW_RETURN_NOT_OK(builder.Finish(&array));
+  return array;
+}
+
+arrow::Result<std::shared_ptr<arrow::RecordBatch>> GetSampleRecordBatch(
+    const arrow::ArrayVector array_vector, const arrow::FieldVector& field_vector) {
+  std::shared_ptr<arrow::RecordBatch> record_batch;
+  ARROW_ASSIGN_OR_RAISE(auto struct_result,
+                        arrow::StructArray::Make(array_vector, field_vector));
+  return record_batch->FromStructArray(struct_result);
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+  std::shared_ptr<arrow::Table> table;
+
+  auto field_vector = {
+      arrow::field("a", arrow::int64()), arrow::field("x", arrow::int64()),
+      arrow::field("y", arrow::int64()), arrow::field("z", arrow::int64()),
+      arrow::field("b", arrow::boolean())};
+
+  ARROW_ASSIGN_OR_RAISE(auto int_array,
+                        GetArrayDataSample<arrow::Int64Type>({1, 2, 3, 4, 5, 6}));
+  ARROW_ASSIGN_OR_RAISE(auto x,
+                        GetArrayDataSample<arrow::Int64Type>({21, 22, 23, 24, 25, 26}));
+  ARROW_ASSIGN_OR_RAISE(auto y,
+                        GetArrayDataSample<arrow::Int64Type>({31, 32, 33, 34, 35, 36}));
+  ARROW_ASSIGN_OR_RAISE(auto z,
+                        GetArrayDataSample<arrow::Int64Type>({41, 42, 43, 44, 45, 46}));
+  ARROW_ASSIGN_OR_RAISE(auto bool_array, GetArrayDataSample<arrow::BooleanType>(
+                                             {false, true, false, true, true, false}));
+
+  auto schema = arrow::schema(field_vector);
+  auto data_vector = {int_array, x, y, z, bool_array};
+
+  table = arrow::Table::Make(schema, data_vector, 6);
+
+  return table;
+}
+
+class UDFOptionsType : public cp::FunctionOptionsType {
+  const char* type_name() const override { return "UDFOptionsType"; }
+  std::string Stringify(const cp::FunctionOptions&) const override {
+    return "UDFOptionsType";
+  }
+  bool Compare(const cp::FunctionOptions&, const cp::FunctionOptions&) const override {
+    return true;
+  }
+  std::unique_ptr<cp::FunctionOptions> Copy(const cp::FunctionOptions&) const override;

Review comment:
       Will include a relative line number after following reviews. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] vibhatha commented on a change in pull request #12590: ARROW-15639 [C++][Python] UDF Scalar Function Implementation

Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12590:
URL: https://github.com/apache/arrow/pull/12590#discussion_r837638681



##########
File path: python/examples/udf/udf_example.py
##########
@@ -0,0 +1,352 @@
+from typing import List

Review comment:
       Of course, we can. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org