You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "krcrouse (via GitHub)" <gi...@apache.org> on 2023/03/31 07:22:41 UTC
[GitHub] [arrow] krcrouse opened a new pull request, #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
krcrouse opened a new pull request, #13126:
URL: https://github.com/apache/arrow/pull/13126
This PR addresses both the JIRA issue cited (pre-generate pyarrow.compute) and also a dev thread that suggests creating the ability to add in python docs for functions that inherit from the Arrow C++ would greatly improve the readability for python users.
There are still a few things to work out, such as where in the build process to generate the code and whether a version of the generated code should be checked into version control or not, but @pitrou suggested opening the PR to field comments from developers.
Major points:
- creates `python/docs/additions` tree where the reStructrued text docs that include the sections to overwrite. Using raw reSt so that code block examples can be tested using doctest - see the README for more verbose details
- `pyarrow.docutils` (or maybe should be _docutils) provides functions to processes `python/docs/additions` and return a data structure of the components per function.
- `python/scripts/generate_sources.py` uses `pyarrow.docutils` and writes out the code for the compute functions in `pyarrow/generated/compute.py`. All of the logic from the release-branch `pyarrow.compute` module that dynamically generated the compute functions has been moved to this script.
- I didn't check the generated file into the repo because I generally do not include generated files that would be generated by the build process should be in source control, but I realize there are other perspectives on this
- `pyarrow.compute` now imports from `pyarrow.generated.compute` for all of the autogenerated compute bindings. Override and custom functions are still defined here.
- The old `pyarrow._compute_docstrings` is gone because its purpose is subsumed in the above.
- I've updated the tests so that they work with the above changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1490654907
Closing because it has been untouched for a while, in case it's still relevant feel free to reopen and move it forward 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1228497759
@krcrouse thanks for the updates! And sorry for the slow follow-up on your responses.
> > As far as I understand, by using the full ability of docutils, we can override _any_ section of the docstring ...
> > Do we think we will really need that full ability? Or it might in almost all cases be sufficient to just append content?
>
> I think we would want both options. Since the default documentation is pulling from the C++ library code, I think you could browse the current [generated documentation](https://arrow.apache.org/docs/python/api/compute.html) and see sections that are not useful and could be entirely overwritten. I also think the hybrid approach of creating default documentation with options to append and/or overwrite is best because it will pull in changes to the core C++ function interface automatically while preserving the manually provided improved pythonic documentation.
(yes, to be clear my question about the need for docutils was about this (do we think appending is sufficient, or do we want the more fine grained replacement), and not to question to use of restructuredtext)
I am personally still a bit hesitant to go the full docutils way here. I certainly see the value of the flexibility it provides, but it also does introduce quite some additional code that needs to be maintained for this. And for now, all the doc additions are pure "append" ones, which could be implemented with much less code (although I know it's the goal to expand the set of doc additions, of course).
> Take, for example, the parameter definitions of [`replace_with_mask`](https://arrow.apache.org/docs/python/generated/pyarrow.compute.replace_with_mask.html#pyarrow.compute.replace_with_mask) - in the context of the still not terribly verbose description, the parameter types are incorrect and the explanation of each parameter is useless ('Argument to compute function.")
Yes, fully agreed that's basically useless. Those are dummy auto-generated on the python side, and the more general solution might be to actually start including argument descriptions in the C++ docs, so we can pull that as well into the python docstrings. There is a TODO about this (cc @pitrou):
https://github.com/apache/arrow/blob/b832853ba62171d5fe5077681083fc6ea49bfd44/cpp/src/arrow/compute/function.h#L132-L135
Although of course, if only Python would make use of those extra descriptions, we could maybe as well keep them on the python side ..
(to be clear, I am not saying that we certainly don't want the more advanced docutils-based approach, but some input from others would be welcome)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by "krcrouse (via GitHub)" <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1496251701
> @krcrouse sorry for letting this slip my mind. I will try to take a look at the latest version and your last comment shortly.
@jorisvandenbossche, sure let me know if this is something that you would like to pursue. I haven't updated it in quite a while since there hasn't been much interest but I'd be willing to do so if there's a desire to incorporate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1179888233
@jorisvandenbossche, I'll wait for some more guidance from you all and just respond to your comments inline for the moment.
> * I think we will probably want to commit the generated code to the repo (and in any case, doing that for now might also make it easier to review (although you already linked the file above as well)).
That makes sense and probably helps in the overall structure.
> * Can you explain a bit more the need for `pyarrow.docutils`?
docutils is used to process the reStructured text appendices so that they can be merged with the autogenerated docs. In this model, I propose using reSt for the function documentation additions because it's established, it's testable, and contributors will at least understand what it is even if they're not proficient in it.
If your question is more about "the need for it as a required module" - If we include the generated files then the `docutils` requirement only needs to be a build dependency in order to generate the actual code files. It does not need to be included as a requirement for the actual package for end users.
> (we should also make it a private module, or put in eg python/scripts or so?)
Agreed - it should be a private module.
> As far as I understand, by using the full ability of docutils, we can override _any_ section of the docstring ...
> Do we think we will really need that full ability? Or it might in almost all cases be sufficient to just append content?
I think we would want both options. Since the default documentation is pulling from the C++ library code, I think you could browse the current [generated documentation](https://arrow.apache.org/docs/python/api/compute.html) and see sections that are not useful and could be entirely overwritten. I also think the hybrid approach of creating default documentation with options to append and/or overwrite is best because it will pull in changes to the core C++ function interface automatically while preserving the manually provided improved pythonic documentation.
Take, for example, the parameter definitions of [`replace_with_mask`](https://arrow.apache.org/docs/python/generated/pyarrow.compute.replace_with_mask.html#pyarrow.compute.replace_with_mask) - in the context of the still not terribly verbose description, the parameter types are incorrect and the explanation of each parameter is useless ('Argument to compute function."). I don't have the code in front of me at the moment, but having implemented this function in pyarrow 7.0, there are several oddities in how these parameters are handled that should be in the param descriptions and not merely as an appended paragraph at the bottom.
> Small nit: this doesn't need to be a public module, so maybe something like `_compute_generated.py` instead?
> * > Using raw reSt so that code block examples can be tested using doctest
Agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on a diff in pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
amol- commented on code in PR #13126:
URL: https://github.com/apache/arrow/pull/13126#discussion_r917935809
##########
python/scripts/generate_sources.py:
##########
@@ -0,0 +1,414 @@
+import pyarrow as pa
+from pyarrow import docutils as arrowdoc
+from pyarrow.vendored import docscrape
+
+from pyarrow._compute import ( # noqa
+ Function,
+ FunctionOptions,
+ FunctionRegistry,
+ HashAggregateFunction,
+ HashAggregateKernel,
+ Kernel,
+ ScalarAggregateFunction,
+ ScalarAggregateKernel,
+ ScalarFunction,
+ ScalarKernel,
+ VectorFunction,
+ VectorKernel,
+ # Option classes
+ ArraySortOptions,
+ AssumeTimezoneOptions,
+ CastOptions,
+ CountOptions,
+ DayOfWeekOptions,
+ DictionaryEncodeOptions,
+ ElementWiseAggregateOptions,
+ ExtractRegexOptions,
+ FilterOptions,
+ IndexOptions,
+ JoinOptions,
+ MakeStructOptions,
+ MapLookupOptions,
+ MatchSubstringOptions,
+ ModeOptions,
+ NullOptions,
+ PadOptions,
+ PartitionNthOptions,
+ QuantileOptions,
+ RandomOptions,
+ ReplaceSliceOptions,
+ ReplaceSubstringOptions,
+ RoundOptions,
+ RoundTemporalOptions,
+ RoundToMultipleOptions,
+ ScalarAggregateOptions,
+ SelectKOptions,
+ SetLookupOptions,
+ SliceOptions,
+ SortOptions,
+ SplitOptions,
+ SplitPatternOptions,
+ StrftimeOptions,
+ StrptimeOptions,
+ StructFieldOptions,
+ TakeOptions,
+ TDigestOptions,
+ TrimOptions,
+ Utf8NormalizeOptions,
+ VarianceOptions,
+ WeekOptions,
+ # Functions
+ call_function,
+ function_registry,
+ get_function,
+ list_functions,
+ _group_by,
+ # Expressions
+ Expression,
+)
+import sys
+import os
+import inspect
+import warnings
+from inspect import Parameter
+from collections import namedtuple
+from textwrap import indent, dedent
+
+# Avoid clashes with Python keywords
+function_name_rewrites = {'and': 'and_', 'or': 'or_'}
+
+def _get_arg_names(func):
+ return func._doc.arg_names
+
+_OptionsClassDoc = namedtuple('_OptionsClassDoc', ('params',))
+
+def _scrape_options_class_doc(options_class):
+ if not options_class.__doc__:
+ return None
+ doc = docscrape.NumpyDocString(options_class.__doc__)
+ return _OptionsClassDoc(doc['Parameters'])
+
+
+def _get_options_class(func):
+ class_name = func._doc.options_class
+ if not class_name:
+ return None
+ try:
+ return globals()[class_name]
+ except KeyError:
+ warnings.warn("Python binding for {} not exposed"
+ .format(class_name), RuntimeWarning)
+ return None
+
+
+def generate_compute_function_doc(exposed_name, func, options_class, custom_overrides = None):
+ """ Create the documentation for functions defined in Arrow C++.
+
+ Args:
+ exposed_name: The name of the function.
+ func: The cython function that connects to Arrow C++.
+ options_class: The class object for the options.
+ custom_overrides: Custom doc overrides as processed by pyarrow.docutils from the `python/docs/additions/compute` directory.
+ Returns:
+ str: The docstring to set the documentation for the pyarrow.compute function.
+ """
+
+ cpp_doc = func._doc
+
+ if not custom_overrides:
+ custom_overrides = {}
+
+ docstring = ""
+
+ # 1. One-line summary
+ summary = cpp_doc.summary
+ if not summary:
+ arg_str = "arguments" if func.arity > 1 else "argument"
+ summary = ("Call compute function {!r} with the given {}"
+ .format(func.name, arg_str))
+
+ docstring += f"{summary}.\n\n"
+
+ # 2.a. Multi-line description
+ if 'description' in custom_overrides and custom_overrides['description']:
+ docstring += custom_overrides['description'] + "\n\n"
+ elif cpp_doc.description:
+ docstring += cpp_doc.description + "\n\n"
+
+ # 2.b. If "details" are provided in the override, add them to the description block
+ if 'details' in custom_overrides and custom_overrides['details']:
+ docstring += "\n\n".join(custom_overrides['details']) + "\n\n"
+
+ # 3. Parameter description
+ docstring += "Parameters\n----------\n"
+
+ if custom_overrides and 'parameters' in custom_overrides:
+ custom_params = custom_overrides['parameters']
+ else:
+ custom_params = {}
+
+ # 3a. Compute function parameters
+ arg_names = _get_arg_names(func)
+ for arg_name in arg_names:
+ if arg_name in custom_params:
+ custom_arg = custom_params[arg_name]
+ else:
+ custom_arg = {}
+
+ if 'classifier' in custom_arg:
+ arg_type = custom_arg['classifier']
+ elif func.kind in ('vector', 'scalar_aggregate'):
+ arg_type = 'Array-like'
+ else:
+ arg_type = 'Array-like or scalar-like'
+ docstring += f"{arg_name} : {arg_type}\n"
+
+ if 'definition' in custom_arg:
+ docstring += f" {custom_arg['definition']}\n"
+ else:
+ docstring += " Argument to compute function.\n"
+
+ # 3b. Compute function option values
+ if options_class is not None:
+ options_class_doc = _scrape_options_class_doc(options_class)
+ if options_class_doc:
+ for p in options_class_doc.params:
+ if custom_overrides and 'parameters' in custom_overrides and p.name in custom_overrides['parameters']:
+ custom_args = custom_overrides['parameters'][p.name]
+ else:
+ custom_args = {}
+
+ if 'type' in custom_args:
+ docstring += f"{p.name} : {custom_args['type']}\n"
+ else:
+ docstring += f"{p.name} : {p.type}\n"
+
+ if 'definition' in custom_args:
+ docstring += f" {custom_args['definition']}"
+ else:
+ for s in p.desc:
+ docstring += f" {s}\n"
+ else:
+ warnings.warn(f"Options class {options_class.__name__} "
+ f"does not have a docstring", RuntimeWarning)
+ options_sig = inspect.signature(options_class)
+ for p in options_sig.parameters.values():
+ docstring += dedent("""\
+ {0} : optional
+ Parameter for {1} constructor. Either `options`
+ or `{0}` can be passed, but not both at the same time.
+ """.format(p.name, options_class.__name__))
+ docstring += dedent(f"""\
+ options : pyarrow.compute.{options_class.__name__}, optional
+ Alternative way of passing options.
+ """)
+
+ docstring += dedent("""\
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.\n\n""")
+
+ # 4. Compute return type
+ if 'return_type' in custom_overrides:
+ return_string = "Returns\n-------\n"
+ for retval, retdesc in custom_overrides['return_type']:
+ return_string += f"{retval}\n {retdesc}\n"
+ docstring += return_string
+
+ # 5. Note about the C++ function
+ docstring += f"See Also\n--------\nThe `{func.name}` compute function in the Arrow C++ library."
+
+ # 6. Custom addition (e.g. examples)
+ if 'examples' in custom_overrides:
+ docstring += "\n\nExamples\n--------\n" + "\n\n".join(custom_overrides['examples'])
+
+ return(docstring)
+
+
+def generate_function_def(name, cpp_name, func, arity, custom_overrides = None):
+ """ Create the function definition for the pyarrow.compute function.
+
+ Args:
+ name: The name of the function.
+ cpp_name: The name of the Arrow C++ function, which might differ slightly from the name parameter.
+ func: The cython function that connects to Arrow C++.
+ arity: The number of non-option arguments to the function.
+ custom_overrides: Custom doc overrides as processed by pyarrow.docutils from the `python/docs/additions/compute` directory, which are passed on to the `generate_compute_function_doc` function.
+ Returns:
+ str: The generated function definition, in string format.
+
+ """
+
+ # prepare args
+ all_params = []
+ # required options
+ options_required = func._doc.options_required
+
+ argnames = _get_arg_names(func)
+ if argnames and argnames[-1].startswith('*'):
+ var_argname = argnames.pop().lstrip('*')
+ else:
+ var_argname = None
+
+ for argname in argnames:
+ all_params.append(Parameter(argname, Parameter.POSITIONAL_ONLY))
+
+ if var_argname:
+ all_params.append(Parameter(var_argname, Parameter.VAR_POSITIONAL))
+ argnames.append('*' + var_argname)
+
+ options_class = _get_options_class(func)
+ options_class_name = 'pyarrow._compute.' + func._doc.options_class
+ option_params = []
+ if options_class is not None:
+ options_sig = inspect.signature(options_class)
+ for paramname, paramdef in options_sig.parameters.items():
+ assert paramdef.kind in (Parameter.POSITIONAL_OR_KEYWORD,
+ Parameter.KEYWORD_ONLY)
+ if var_argname:
+ # Cannot have a positional argument after a *args
+ paramdef = paramdef.replace(kind=Parameter.KEYWORD_ONLY)
+ if paramdef.default == inspect._empty:
+ paramdef = paramdef.replace(default = None)
+ all_params.append(paramdef)
+ option_params.append(paramname)
+ all_params.append(Parameter("options", Parameter.KEYWORD_ONLY,
+ default=None))
+ all_params.append(Parameter("memory_pool", Parameter.KEYWORD_ONLY,
+ default=None))
+
+ #funcparams = make_function_signature(arg_names, var_arg_names, options_class)
+
+ funcdoc = generate_compute_function_doc(name, func, options_class, custom_overrides)
+ funcdoc = indent(funcdoc, " " * 12, lambda l: l != "\n")
+
+ if len(argnames) == 1:
+ #argstring = f'( {argnames[0]}, )'
+ argstring = f'[ {argnames[0]} ]'
+ else:
+ #argstring = f"( {', '.join(argnames)} )"
+ argstring = f"[ {', '.join(argnames)} ]"
+
+ full_signature = inspect.Signature(all_params)
+ if not options_class:
+ # -- create the expression handling the expression if there is a regular argument
+ if len(argnames) > 2 or (len(argnames) and not var_argname):
+ expression_clause = f'''\
+
+ if isinstance({argnames[0]}, Expression):
+ return Expression._call('{name}', {argstring})
+ '''
+ else:
+ expression_clause = ""
+
+ function_text = dedent(f'''\
+ def {name}{full_signature}:
+ """\n{funcdoc}
+ """
+ func = pyarrow._compute.get_function('{cpp_name}')
+ {expression_clause}
+ return(
+ func.call({argstring}, memory_pool=memory_pool)
+ )
+ ''')
+ else:
+ # here we need to create the kwargs param substring for hte _handle_options function
+ if arity is Ellipsis or len(argnames) <= arity:
+ option_args = ''
+ else:
+ option_args = ', '.join(argnames[arity:]) + ','
+
+ if len(argnames) > 2 or (len(argnames) and not var_argname):
+ expression_clause = f'''\
+
+ if isinstance({argnames[0]}, Expression):
+ return Expression._call('{name}', {argstring}, options)
+ '''
+ else:
+ expression_clause = ""
+
+
+ function_text = dedent(f'''\
+ def {name}{full_signature}:
+ """\n{funcdoc}
+ """
+
+ options = _handle_options('{name}', {options_class_name}, options,
+ ({option_args}), {', '.join(opt + '=' + opt for opt in option_params)})
+ func = pyarrow._compute.get_function('{cpp_name}')
+ {expression_clause}
+ return(
+ func.call( {argstring}, options, memory_pool)
+ )
+ ''')
+
+ return(function_text)
+
+
+def write_compute_file(output_path):
+ """
+ Write the full set of generated functions to the output_path.
+
+ Note that, in practice, while this generates core functions for all of the Arrow C++ compute functions, the compute functions may be overriden in the `pyarrow/compute.py` file.
+
+ Args:
+ output_path: The full path to the file to write the compute functions to.
+ """
+ g = globals()
+ reg = function_registry()
+
+ doc_overrides = arrowdoc.parse_directory(os.path.dirname(arrowdoc.__file__) + "/../docs/additions/compute")
+
+ function_defs = []
+ for cpp_name in reg.list_functions():
+ name = function_name_rewrites.get(cpp_name, cpp_name)
+
+ func = reg.get_function(cpp_name)
+
+ if func.kind == "hash_aggregate":
+ # Hash aggregate functions are not callable,
+ # so let's not expose them at module level.
+ continue
+
+ function_def = generate_function_def(name, cpp_name, func, func.arity, doc_overrides.get(name, None))
+ function_defs.append(function_def)
+
+ with open(output_path, 'w') as fh:
+ fh.write(dedent(f"""\
+ import pyarrow
Review Comment:
We need an header here, something that mentions this is an autogenerated file and to not modify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1228457138
Hi @jorisvandenbossche and @pitrou,
I made a number of additions so that the autogenerated _compute.py code conformed with flake8, since it doesn't distinguish between hand written and autogenerated code. I also updated the generated function signatures to be compliant with Python 3.7.
Please let me know how you would like to move forward on this / if you do.
If I'm reading the workflow output correctly, I think the only issue right now is that `archery` is failing due to the lack of licenses in the ReSTructured text files, and I am not sure how that can be added as I don't know of any standard for adding licenses to ReST (output from workflow failure below)
```bash
INFO:archery:Running Docker linter
apache-rat license violation: python/docs/additions/compute/all.rst
apache-rat license violation: python/docs/additions/compute/any.rst
apache-rat license violation: python/docs/additions/compute/count.rst
apache-rat license violation: python/docs/additions/compute/count_distinct.rst
apache-rat license violation: python/docs/additions/compute/filter.rst
apache-rat license violation: python/docs/additions/compute/index.rst
apache-rat license violation: python/docs/additions/compute/indices_nonzero.rst
apache-rat license violation: python/docs/additions/compute/mode.rst
apache-rat license violation: python/docs/additions/test_example.rst
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1236365597
@jorisvandenbossche ,
Here's the run down of the most recent push to the branch, which includes updates from upstream as of the end of last week:
Quick Summary Points (many that condense prior one-off comments):
* Based on your comment, I moved `python/pyarrow/_rstutils.py` into `python/scripts/lib/arrowdoc.py` as it is not needed. I don't love a sub-lib of `scripts`, but it's at least straightforward to then have the `generate_sources.py` script reference it.
* As I had replied earlier, there should eventually have a step added to the build or CI process that verifies the committed version of `python/pyarrow/_compute_generated.py` is the same thing that is output from the `generate_sources.py` script.
* The `archery lint --rat` test is failing because the rst files in `python/docs/additions/compute/` don't have license text and I do not know how do that in a sensible way and don't see it in docs or examples.
As for the points you brought up about `docutils` - I wonder if you are misunderstanding its role and how I use it in the PR. It's probably more straightforward with the reorganization in this round. Now, the `docutils` modules is only used in the `scripts/lib/arrodoc.py` file, and there it is only used to parse the reST files in `python/docs/additions/compute/` into the document tree that is then merged/replaced/appended to the docs pulled from the C++ libraries. I did add it to the `requirements-build` and `requirements-wheel-build` files for the PR, but neither of those are really accurate. It's required solely to run the script, and so I felt that it would need to go into some sort of "requirements," but if you're not regenerating the compute functions for python, it's not required for anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1232287180
> Would we need a check in CI that ensures the generated compute file is up-to-date? (I am not directly sure how we do that in other places with generated files)
Yes. One straightforward (though arguably not terribly elegant) way to check this would be to have the CI task run `python scripts/generate_sources.py` with a flag to redirect output and ensure that the output was exactly the same as the `python/pyarrow/_compute_generated.py` file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1228499621
Would we need a check in CI that ensures the generated compute file is up-to-date? (I am not directly sure how we do that in other places with generated files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on a diff in pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on code in PR #13126:
URL: https://github.com/apache/arrow/pull/13126#discussion_r959010642
##########
python/docs/additions/compute/all.rst:
##########
@@ -0,0 +1,24 @@
+all
+===
+
+Examples
+--------
+
+.. code-block:: python
Review Comment:
So it does! I didn't realize that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on a diff in pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on code in PR #13126:
URL: https://github.com/apache/arrow/pull/13126#discussion_r959021762
##########
python/pyarrow/_compute_generated.py:
##########
@@ -0,0 +1,9381 @@
+# File GENERATED by scripts/generate_sources.py - DO NOT EDIT.
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyarrow
+import pyarrow._compute
+from pyarrow._compute import Expression
+
+
+def _handle_options(name, options_class, options, args, **kwargs):
+ if options is not None:
+ if isinstance(options, dict):
+ return options_class(**options)
+ elif isinstance(options, options_class):
+ return options
+ raise TypeError(
+ "Function {!r} expected a {} parameter, got {}"
+ .format(name, options_class, type(options)))
+
+ if args or kwargs:
+ # Note: This check is no longer permissable
+ # Generating function code with real signatures means that
+ # All of the keyword arguments have default values, and so
+ # this would always be true. As the default for the options object
+ # is always false, the options object takes precedence if provided.
+ #
+ # if options is not None:
+ # raise TypeError(
+ # "Function {!r} called with both an 'options' argument "
+ # "and additional arguments"
+ # .format(name))
+
+ return options_class(*args, **kwargs)
+
+ return None
+
+
+def abs(x, *, memory_pool=None):
+ """Calculate the absolute value of the argument element-wise.
+
+ Results will wrap around on integer overflow.
+ Use function "abs_checked" if you want overflow
+ to return an error.
+
+ This wraps the "abs" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('abs')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'abs',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def abs_checked(x, *, memory_pool=None):
+ """Calculate the absolute value of the argument element-wise.
+
+ This function returns an error on overflow. For a variant that
+ doesn't fail on overflow, use function "abs".
+
+ This wraps the "abs_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('abs_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'abs_checked',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def acos(x, *, memory_pool=None):
+ """Compute the inverse cosine.
+
+ NaN is returned for invalid input values;
+ to raise an error instead, see "acos_checked".
+
+ This wraps the "acos" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('acos')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'acos',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def acos_checked(x, *, memory_pool=None):
+ """Compute the inverse cosine.
+
+ Invalid input values raise an error;
+ to return NaN instead, see "acos".
+
+ This wraps the "acos_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('acos_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'acos_checked',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def add(x, y, *, memory_pool=None):
+ """Add the arguments element-wise.
+
+ Results will wrap around on integer overflow.
+ Use function "add_checked" if you want overflow
+ to return an error.
+
+ This wraps the "add" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('add')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'add',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def add_checked(x, y, *, memory_pool=None):
+ """Add the arguments element-wise.
+
+ This function returns an error on overflow. For a variant that
+ doesn't fail on overflow, use function "add".
+
+ This wraps the "add_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('add_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'add_checked',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def all(array, *, skip_nulls=True, min_count=1, options=None,
+ memory_pool=None):
+ """Test whether all elements in a boolean array evaluate to true.
+
+ Null values are ignored by default.
+ If the `skip_nulls` option is set to false, then Kleene logic is used.
+ See "kleene_and" for more details on Kleene logic.
+
+ This wraps the "all" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ skip_nulls : bool, default True
+ Whether to skip (ignore) nulls in the input.
+ If False, any null in the input forces the output to null.
+ min_count : int, default 1
+ Minimum number of non-null values in the input. If the number
+ of non-null values is below `min_count`, the output is null.
+ options : pyarrow.compute.ScalarAggregateOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+
+
+
+ Examples
+ --------
+ >>> import pyarrow as pa
+ >>> import pyarrow.compute as pc
+ >>> arr = pa.array([True, True, None, False, True])
+ >>> pc.all(arr)
+ <pyarrow.BooleanScalar: False>
+ >>> arr = pa.array([True, True, None, True, None, None])
+ >>> pc.all(arr)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.all(arr, skip_nulls = False)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.all(arr, min_count = 4)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.all(arr, min_count = 10)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.all(arr, min_count = 2)
+ <pyarrow.BooleanScalar: True>
+ """
+
+ _computed_options = _handle_options(
+ 'all',
+ pyarrow._compute.ScalarAggregateOptions,
+ options,
+ (),
+ skip_nulls=skip_nulls,
+ min_count=min_count
+ )
+ func = pyarrow._compute.get_function('all')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'all',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def and_(x, y, *, memory_pool=None):
+ """Logical 'and' boolean values.
+
+ When a null is encountered in either input, a null is output.
+ For a different null behavior, see function "and_kleene".
+
+ This wraps the "and" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def and_kleene(x, y, *, memory_pool=None):
+ """Logical 'and' boolean values (Kleene logic).
+
+ This function behaves as follows with nulls:
+
+ - true and null = null
+ - null and true = null
+ - false and null = false
+ - null and false = false
+ - null and null = null
+
+ In other words, in this context a null value really means "unknown",
+ and an unknown value 'and' false is always false.
+ For a different null behavior, see function "and".
+
+ This wraps the "and_kleene" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and_kleene')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_kleene',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def and_not(x, y, *, memory_pool=None):
+ """Logical 'and not' boolean values.
+
+ When a null is encountered in either input, a null is output.
+ For a different null behavior, see function "and_not_kleene".
+
+ This wraps the "and_not" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and_not')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_not',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def and_not_kleene(x, y, *, memory_pool=None):
+ """Logical 'and not' boolean values (Kleene logic).
+
+ This function behaves as follows with nulls:
+
+ - true and null = null
+ - null and false = null
+ - false and null = false
+ - null and true = false
+ - null and null = null
+
+ In other words, in this context a null value really means "unknown",
+ and an unknown value 'and not' true is always false, as is false
+ 'and not' an unknown value.
+ For a different null behavior, see function "and_not".
+
+ This wraps the "and_not_kleene" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and_not_kleene')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_not_kleene',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def any(array, *, skip_nulls=True, min_count=1, options=None,
+ memory_pool=None):
+ """Test whether any element in a boolean array evaluates to true.
+
+ Null values are ignored by default.
+ If the `skip_nulls` option is set to false, then Kleene logic is used.
+ See "kleene_or" for more details on Kleene logic.
+
+ This wraps the "any" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ skip_nulls : bool, default True
+ Whether to skip (ignore) nulls in the input.
+ If False, any null in the input forces the output to null.
+ min_count : int, default 1
+ Minimum number of non-null values in the input. If the number
+ of non-null values is below `min_count`, the output is null.
+ options : pyarrow.compute.ScalarAggregateOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+
+
+
+ Examples
+ --------
+ >>> import pyarrow as pa
+ >>> import pyarrow.compute as pc
+ >>> arr = pa.array([True, True, None, False, True])
+ >>> pc.any(arr)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = 4)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = len(arr))
+ <pyarrow.BooleanScalar: None>
+ >>> arr = pa.array([False, False, None, False, True])
+ >>> pc.any(arr)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = 2)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = len(arr))
+ <pyarrow.BooleanScalar: None>
+ >>> pc.any(arr, skip_nulls = False)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any([False,None], skip_nulls = False)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.any([False,None], skip_nulls = True)
+ <pyarrow.BooleanScalar: False>
+ """
+
+ _computed_options = _handle_options(
+ 'any',
+ pyarrow._compute.ScalarAggregateOptions,
+ options,
+ (),
+ skip_nulls=skip_nulls,
+ min_count=min_count
+ )
+ func = pyarrow._compute.get_function('any')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'any',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def approximate_median(array, *, skip_nulls=True, min_count=1, options=None,
+ memory_pool=None):
+ """Approximate median of a numeric array with T-Digest algorithm.
+
+ Nulls and NaNs are ignored.
+ A null scalar is returned if there is no valid data point.
+
+ This wraps the "approximate_median" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ skip_nulls : bool, default True
+ Whether to skip (ignore) nulls in the input.
+ If False, any null in the input forces the output to null.
+ min_count : int, default 1
+ Minimum number of non-null values in the input. If the number
+ of non-null values is below `min_count`, the output is null.
+ options : pyarrow.compute.ScalarAggregateOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'approximate_median',
+ pyarrow._compute.ScalarAggregateOptions,
+ options,
+ (),
+ skip_nulls=skip_nulls,
+ min_count=min_count
+ )
+ func = pyarrow._compute.get_function('approximate_median')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'approximate_median',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def array_filter(array, selection_filter, null_selection_behavior='drop', *,
+ options=None, memory_pool=None):
+ """Filter with a boolean selection filter.
+
+ The output is populated with values from the input `array` at positions
+ where the selection filter is non-zero. Nulls in the selection filter
+ are handled based on FilterOptions.
+
+ This wraps the "array_filter" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ selection_filter : Array-like
+ Argument to compute function.
+ null_selection_behavior : str, default "drop"
+ How to handle nulls in the selection filter.
+ Accepted values are "drop", "emit_null".
+ options : pyarrow.compute.FilterOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'array_filter',
+ pyarrow._compute.FilterOptions,
+ options,
+ (),
+ null_selection_behavior=null_selection_behavior
+ )
+ func = pyarrow._compute.get_function('array_filter')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'array_filter',
+ [array, selection_filter],
+ _computed_options
+ )
+
+ return (
+ func.call([array, selection_filter], _computed_options, memory_pool)
+ )
+
+
+def array_sort_indices(array, order='ascending', *, null_placement='at_end',
+ options=None, memory_pool=None):
+ """Return the indices that would sort an array.
+
+ This function computes an array of indices that define a stable sort
+ of the input array. By default, Null values are considered greater
+ than any other value and are therefore sorted at the end of the array.
+ For floating-point types, NaNs are considered greater than any
+ other non-null value, but smaller than null values.
+
+ The handling of nulls and NaNs can be changed in ArraySortOptions.
+
+ This wraps the "array_sort_indices" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ order : str, default "ascending"
+ Which order to sort values in.
+ Accepted values are "ascending", "descending".
+ null_placement : str, default "at_end"
+ Where nulls in the input should be sorted.
+ Accepted values are "at_start", "at_end".
+ options : pyarrow.compute.ArraySortOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'array_sort_indices',
+ pyarrow._compute.ArraySortOptions,
+ options,
+ (),
+ order=order,
+ null_placement=null_placement
+ )
+ func = pyarrow._compute.get_function('array_sort_indices')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'array_sort_indices',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def array_take(array, indices, *, boundscheck=True, options=None,
+ memory_pool=None):
+ """Select values from an array based on indices from another array.
+
+ The output is populated with values from the input array at positions
+ given by `indices`. Nulls in `indices` emit null in the output.
+
+ This wraps the "array_take" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ indices : Array-like
+ Argument to compute function.
+ boundscheck : boolean, default True
+ Whether to check indices are within bounds. If False and an
+ index is out of boundes, behavior is undefined (the process
+ may crash).
+ options : pyarrow.compute.TakeOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'array_take',
+ pyarrow._compute.TakeOptions,
+ options,
+ (),
+ boundscheck=boundscheck
+ )
+ func = pyarrow._compute.get_function('array_take')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'array_take',
+ [array, indices],
+ _computed_options
+ )
+
+ return (
+ func.call([array, indices], _computed_options, memory_pool)
+ )
+
+
+def ascii_capitalize(strings, *, memory_pool=None):
+ """Capitalize the first character of ASCII input.
+
+ For each string in `strings`, return a capitalized version.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_capitalize" instead.
+
+ This wraps the "ascii_capitalize" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_capitalize')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_capitalize',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_center(strings, width=None, padding=' ', *, options=None,
+ memory_pool=None):
+ """Center strings by padding with a given character.
+
+ For each string in `strings`, emit a centered string by padding both
+ sides
+ with the given ASCII character.
+ Null values emit null.
+
+ This wraps the "ascii_center" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ width : int
+ Desired string length.
+ padding : str, default " "
+ What to pad the string with. Should be one byte or codepoint.
+ options : pyarrow.compute.PadOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_center',
+ pyarrow._compute.PadOptions,
+ options,
+ (),
+ width=width,
+ padding=padding
+ )
+ func = pyarrow._compute.get_function('ascii_center')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_center',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_is_alnum(strings, *, memory_pool=None):
+ """Classify strings as ASCII alphanumeric.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of alphanumeric ASCII characters. Null strings emit
+ null.
+
+ This wraps the "ascii_is_alnum" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_alnum')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_alnum',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_alpha(strings, *, memory_pool=None):
+ """Classify strings as ASCII alphabetic.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of alphabetic ASCII characters. Null strings emit
+ null.
+
+ This wraps the "ascii_is_alpha" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_alpha')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_alpha',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_decimal(strings, *, memory_pool=None):
+ """Classify strings as ASCII decimal.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of decimal ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_decimal" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_decimal')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_decimal',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_lower(strings, *, memory_pool=None):
+ """Classify strings as ASCII lowercase.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of lowercase ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_lower" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_lower')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_lower',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_printable(strings, *, memory_pool=None):
+ """Classify strings as ASCII printable.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of printable ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_printable" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_printable')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_printable',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_space(strings, *, memory_pool=None):
+ """Classify strings as ASCII whitespace.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of whitespace ASCII characters. Null strings emit
+ null.
+
+ This wraps the "ascii_is_space" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_space')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_space',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_title(strings, *, memory_pool=None):
+ """Classify strings as ASCII titlecase.
+
+ For each string in `strings`, emit true iff the string is title-cased,
+ i.e. it has at least one cased character, each uppercase character
+ follows an uncased character, and each lowercase character follows
+ an uppercase character.
+
+ This wraps the "ascii_is_title" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_title')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_title',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_upper(strings, *, memory_pool=None):
+ """Classify strings as ASCII uppercase.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of uppercase ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_upper" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_upper')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_upper',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_lower(strings, *, memory_pool=None):
+ """Transform ASCII input to lowercase.
+
+ For each string in `strings`, return a lowercase version.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_lower" instead.
+
+ This wraps the "ascii_lower" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_lower')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_lower',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_lpad(strings, width=None, padding=' ', *, options=None,
+ memory_pool=None):
+ """Right-align strings by padding with a given character.
+
+ For each string in `strings`, emit a right-aligned string by prepending
+ the given ASCII character.
+ Null values emit null.
+
+ This wraps the "ascii_lpad" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ width : int
+ Desired string length.
+ padding : str, default " "
+ What to pad the string with. Should be one byte or codepoint.
+ options : pyarrow.compute.PadOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_lpad',
+ pyarrow._compute.PadOptions,
+ options,
+ (),
+ width=width,
+ padding=padding
+ )
+ func = pyarrow._compute.get_function('ascii_lpad')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_lpad',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_ltrim(strings, characters=None, *, options=None, memory_pool=None):
+ """Trim leading characters.
+
+ For each string in `strings`, remove any leading characters
+ from the `characters` option (as given in TrimOptions).
+ Null values emit null.
+ Both the `strings` and the `characters` are interpreted as
+ ASCII; to trim non-ASCII characters, use `utf8_ltrim`.
+
+ This wraps the "ascii_ltrim" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ characters : str
+ Individual characters to be trimmed from the string.
+ options : pyarrow.compute.TrimOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_ltrim',
+ pyarrow._compute.TrimOptions,
+ options,
+ (),
+ characters=characters
+ )
+ func = pyarrow._compute.get_function('ascii_ltrim')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_ltrim',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_ltrim_whitespace(strings, *, memory_pool=None):
+ """Trim leading ASCII whitespace characters.
+
+ For each string in `strings`, emit a string with leading ASCII whitespace
+ characters removed. Use `utf8_ltrim_whitespace` to trim leading Unicode
+ whitespace characters. Null values emit null.
+
+ This wraps the "ascii_ltrim_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_ltrim_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_ltrim_whitespace',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_reverse(strings, *, memory_pool=None):
+ """Reverse ASCII input.
+
+ For each ASCII string in `strings`, return a reversed version.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_reverse" instead.
+
+ This wraps the "ascii_reverse" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_reverse')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_reverse',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_rpad(strings, width=None, padding=' ', *, options=None,
+ memory_pool=None):
+ """Left-align strings by padding with a given character.
+
+ For each string in `strings`, emit a left-aligned string by appending
+ the given ASCII character.
+ Null values emit null.
+
+ This wraps the "ascii_rpad" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ width : int
+ Desired string length.
+ padding : str, default " "
+ What to pad the string with. Should be one byte or codepoint.
+ options : pyarrow.compute.PadOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_rpad',
+ pyarrow._compute.PadOptions,
+ options,
+ (),
+ width=width,
+ padding=padding
+ )
+ func = pyarrow._compute.get_function('ascii_rpad')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_rpad',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_rtrim(strings, characters=None, *, options=None, memory_pool=None):
+ """Trim trailing characters.
+
+ For each string in `strings`, remove any trailing characters
+ from the `characters` option (as given in TrimOptions).
+ Null values emit null.
+ Both the `strings` and the `characters` are interpreted as
+ ASCII; to trim non-ASCII characters, use `utf8_rtrim`.
+
+ This wraps the "ascii_rtrim" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ characters : str
+ Individual characters to be trimmed from the string.
+ options : pyarrow.compute.TrimOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_rtrim',
+ pyarrow._compute.TrimOptions,
+ options,
+ (),
+ characters=characters
+ )
+ func = pyarrow._compute.get_function('ascii_rtrim')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_rtrim',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_rtrim_whitespace(strings, *, memory_pool=None):
+ """Trim trailing ASCII whitespace characters.
+
+ For each string in `strings`, emit a string with trailing ASCII
+ whitespace
+ characters removed. Use `utf8_rtrim_whitespace` to trim trailing Unicode
+ whitespace characters. Null values emit null.
+
+ This wraps the "ascii_rtrim_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_rtrim_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_rtrim_whitespace',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_split_whitespace(strings, *, max_splits=None, reverse=False,
+ options=None,
+ memory_pool=None):
+ """Split string according to any ASCII whitespace.
+
+ Split each string according any non-zero length sequence of ASCII
+ whitespace characters. The output for each string input is a list
+ of strings.
+
+ The maximum number of splits and direction of splitting
+ (forward, reverse) can optionally be defined in SplitOptions.
+
+ This wraps the "ascii_split_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ max_splits : int or None, default None
+ Maximum number of splits for each input value (unlimited if None).
+ reverse : bool, default False
+ Whether to start splitting from the end of each input value.
+ This only has an effect if `max_splits` is not None.
+ options : pyarrow.compute.SplitOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_split_whitespace',
+ pyarrow._compute.SplitOptions,
+ options,
+ (),
+ max_splits=max_splits,
+ reverse=reverse
+ )
+ func = pyarrow._compute.get_function('ascii_split_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_split_whitespace',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_swapcase(strings, *, memory_pool=None):
+ """Transform ASCII input by inverting casing.
+
+ For each string in `strings`, return a string with opposite casing.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_swapcase" instead.
+
+ This wraps the "ascii_swapcase" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_swapcase')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_swapcase',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_title(strings, *, memory_pool=None):
+ """Titlecase each word of ASCII input.
+
+ For each string in `strings`, return a titlecased version.
+ Each word in the output will start with an uppercase character and its
+ remaining characters will be lowercase.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_title" instead.
+
+ This wraps the "ascii_title" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_title')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_title',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_trim(strings, characters=None, *, options=None, memory_pool=None):
+ """Trim leading and trailing characters.
+
+ For each string in `strings`, remove any leading or trailing characters
+ from the `characters` option (as given in TrimOptions).
+ Null values emit null.
+ Both the `strings` and the `characters` are interpreted as
+ ASCII; to trim non-ASCII characters, use `utf8_trim`.
+
+ This wraps the "ascii_trim" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ characters : str
+ Individual characters to be trimmed from the string.
+ options : pyarrow.compute.TrimOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_trim',
+ pyarrow._compute.TrimOptions,
+ options,
+ (),
+ characters=characters
+ )
+ func = pyarrow._compute.get_function('ascii_trim')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_trim',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_trim_whitespace(strings, *, memory_pool=None):
+ """Trim leading and trailing ASCII whitespace characters.
+
+ For each string in `strings`, emit a string with leading and trailing
+ ASCII
+ whitespace characters removed. Use `utf8_trim_whitespace` to trim Unicode
+ whitespace characters. Null values emit null.
+
+ This wraps the "ascii_trim_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_trim_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_trim_whitespace',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_upper(strings, *, memory_pool=None):
+ """Transform ASCII input to uppercase.
+
+ For each string in `strings`, return an uppercase version.
+
+ This function assumes the input is fully ASCII. It it may contain
+ non-ASCII characters, use "utf8_upper" instead.
+
+ This wraps the "ascii_upper" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_upper')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_upper',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def asin(x, *, memory_pool=None):
+ """Compute the inverse sine.
+
+ NaN is returned for invalid input values;
+ to raise an error instead, see "asin_checked".
+
+ This wraps the "asin" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('asin')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'asin',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def asin_checked(x, *, memory_pool=None):
+ """Compute the inverse sine.
+
+ Invalid input values raise an error;
+ to return NaN instead, see "asin".
+
+ This wraps the "asin_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('asin_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'asin_checked',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def assume_timezone(timestamps, timezone=None, *, ambiguous='raise',
+ nonexistent='raise', options=None,
+ memory_pool=None):
+ """Convert naive timestamp to timezone-aware timestamp.
+
+ Input timestamps are assumed to be relative to the timezone given in the
+ `timezone` option. They are converted to UTC-relative timestamps and
+ the output type has its timezone set to the value of the `timezone`
+ option. Null values emit null.
+ This function is meant to be used when an external system produces
+ "timezone-naive" timestamps which need to be converted to
+ "timezone-aware" timestamps. An error is returned if the timestamps
+ already have a defined timezone.
+
+ This wraps the "assume_timezone" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ timestamps : Array-like or scalar-like
+ Argument to compute function.
+ timezone : str
+ Timezone to assume for the input.
+ ambiguous : str, default "raise"
+ How to handle timestamps that are ambiguous in the assumed timezone.
+ Accepted values are "raise", "earliest", "latest".
+ nonexistent : str, default "raise"
+ How to handle timestamps that don't exist in the assumed timezone.
+ Accepted values are "raise", "earliest", "latest".
+ options : pyarrow.compute.AssumeTimezoneOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'assume_timezone',
+ pyarrow._compute.AssumeTimezoneOptions,
+ options,
+ (),
+ timezone=timezone,
+ ambiguous=ambiguous,
+ nonexistent=nonexistent
+ )
+ func = pyarrow._compute.get_function('assume_timezone')
+
+ if isinstance(timestamps, Expression):
+ return Expression._call(
+ 'assume_timezone',
+ [timestamps],
+ _computed_options
+ )
+
+ return (
+ func.call([timestamps], _computed_options, memory_pool)
+ )
+
+
+def atan(x, *, memory_pool=None):
+ """Compute the inverse tangent of x.
+
+ The return value is in the range [-pi/2, pi/2];
+ for a full return range [-pi, pi], see "atan2".
+
+ This wraps the "atan" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('atan')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'atan',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def atan2(y, x, *, memory_pool=None):
+ """Compute the inverse tangent of y/x.
+
+ The return value is in the range [-pi, pi].
+
+ This wraps the "atan2" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ y : Array-like or scalar-like
+ Argument to compute function.
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('atan2')
+
+ if isinstance(y, Expression):
+ return Expression._call(
+ 'atan2',
+ [y, x]
+ )
+
+ return (
+ func.call([y, x], memory_pool=memory_pool)
+ )
+
+
+def binary_join(strings, separator, *, memory_pool=None):
+ """Join a list of strings together with a separator.
+
+ Concatenate the strings in `list`. The `separator` is inserted
+ between each given string.
+ Any null input and any null `list` element emits a null output.
+
+ This wraps the "binary_join" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ separator : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_join')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_join',
+ [strings, separator]
+ )
+
+ return (
+ func.call([strings, separator], memory_pool=memory_pool)
+ )
+
+
+def binary_join_element_wise(*strings, null_handling='emit_null',
+ null_replacement='',
+ options=None,
+ memory_pool=None):
+ """Join string arguments together, with the last argument as separator.
+
+ Concatenate the `strings` except for the last one. The last argument
+ in `strings` is inserted between each given string.
+ Any null separator element emits a null output. Null elements either
+ emit a null (the default), are skipped, or replaced with a given string.
+
+ This wraps the "binary_join_element_wise" compute function in the Arrow
+ C++ library.
+
+ Parameters
+ ----------
+ *strings : Array-like or scalar-like
+ Argument to compute function.
+ null_handling : str, default "emit_null"
+ How to handle null values in the inputs.
+ Accepted values are "emit_null", "skip", "replace".
+ null_replacement : str, default ""
+ Replacement string to emit for null inputs if `null_handling`
+ is "replace".
+ options : pyarrow.compute.JoinOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'binary_join_element_wise',
+ pyarrow._compute.JoinOptions,
+ options,
+ (),
+ null_handling=null_handling,
+ null_replacement=null_replacement
+ )
+ func = pyarrow._compute.get_function('binary_join_element_wise')
+ return (
+ func.call([*strings], _computed_options, memory_pool)
+ )
+
+
+def binary_length(strings, *, memory_pool=None):
+ """Compute string lengths.
+
+ For each string in `strings`, emit its length of bytes.
+ Null values emit null.
+
+ This wraps the "binary_length" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_length')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_length',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def binary_repeat(strings, num_repeats, *, memory_pool=None):
+ """Repeat a binary string.
+
+ For each binary string in `strings`, return a replicated version.
+
+ This wraps the "binary_repeat" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ num_repeats : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_repeat')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_repeat',
+ [strings, num_repeats]
+ )
+
+ return (
+ func.call([strings, num_repeats], memory_pool=memory_pool)
+ )
+
+
+def binary_replace_slice(strings, start=None, stop=None, replacement=None, *,
+ options=None,
+ memory_pool=None):
+ """Replace a slice of a binary string.
+
+ For each string in `strings`, replace a slice of the string defined by
+ `start`
+ and `stop` indices with the given `replacement`. `start` is inclusive
+ and `stop` is exclusive, and both are measured in bytes.
+ Null values emit null.
+
+ This wraps the "binary_replace_slice" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ start : int
+ Index to start slicing at (inclusive).
+ stop : int
+ Index to stop slicing at (exclusive).
+ replacement : str
+ What to replace the slice with.
+ options : pyarrow.compute.ReplaceSliceOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'binary_replace_slice',
+ pyarrow._compute.ReplaceSliceOptions,
+ options,
+ (),
+ start=start,
+ stop=stop,
+ replacement=replacement
+ )
+ func = pyarrow._compute.get_function('binary_replace_slice')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_replace_slice',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def binary_reverse(strings, *, memory_pool=None):
+ """Reverse binary input.
+
+ For each binary string in `strings`, return a reversed version.
+
+ This function reverses the binary data at a byte-level.
+
+ This wraps the "binary_reverse" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_reverse')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_reverse',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_and(x, y, *, memory_pool=None):
+ """Bit-wise AND the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_and" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_and')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_and',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_not(x, *, memory_pool=None):
+ """Bit-wise negate the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_not" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_not')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_not',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_or(x, y, *, memory_pool=None):
+ """Bit-wise OR the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_or" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_or')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_or',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_xor(x, y, *, memory_pool=None):
+ """Bit-wise XOR the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_xor" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_xor')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_xor',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def case_when(cond, *cases, memory_pool=None):
+ """Choose values based on multiple conditions.
+
+ `cond` must be a struct of Boolean values. `cases` can be a mix
+ of scalar and array arguments (of any type, but all must be the
+ same type or castable to a common type), with either exactly one
+ datum per child of `cond`, or one more `cases` than children of
+ `cond` (in which case we have an "else" value).
+
+ Each row of the output will be the corresponding value of the
+ first datum in `cases` for which the corresponding child of `cond`
+ is true, or otherwise the "else" value (if given), or null.
+
+ Essentially, this implements a switch-case or if-else, if-else...
+ statement.
+
+ This wraps the "case_when" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ cond : Array-like or scalar-like
+ Argument to compute function.
+ *cases : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('case_when')
+ return (
+ func.call([cond, *cases], memory_pool=memory_pool)
+ )
+
+
+def cast(input, target_type=None, *, allow_int_overflow=None,
+ allow_time_truncate=None, allow_time_overflow=None,
+ allow_decimal_truncate=None, allow_float_truncate=None,
+ allow_invalid_utf8=None, options=None, memory_pool=None):
+ """Cast values to another data type.
+
+ Behavior when values wouldn't fit in the target type
+ can be controlled through CastOptions.
+
+ This wraps the "cast" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ input : Array-like or scalar-like
+ Argument to compute function.
+ target_type : DataType, optional
+ The PyArrow type to cast to.
+ allow_int_overflow : bool, default False
+ Whether integer overflow is allowed when casting.
+ allow_time_truncate : bool, default False
+ Whether time precision truncation is allowed when casting.
+ allow_time_overflow : bool, default False
+ Whether date/time range overflow is allowed when casting.
+ allow_decimal_truncate : bool, default False
+ Whether decimal precision truncation is allowed when casting.
+ allow_float_truncate : bool, default False
+ Whether floating-point precision truncation is allowed when casting.
+ allow_invalid_utf8 : bool, default False
+ Whether producing invalid utf8 data is allowed when casting.
+ options : pyarrow.compute.CastOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'cast',
+ pyarrow._compute.CastOptions,
+ options,
+ (),
+ target_type=target_type,
+ allow_int_overflow=allow_int_overflow,
+ allow_time_truncate=allow_time_truncate,
+ allow_time_overflow=allow_time_overflow,
+ allow_decimal_truncate=allow_decimal_truncate,
+ allow_float_truncate=allow_float_truncate,
+ allow_invalid_utf8=allow_invalid_utf8
+ )
+ func = pyarrow._compute.get_function('cast')
+
+ if isinstance(input, Expression):
+ return Expression._call(
+ 'cast',
+ [input],
+ _computed_options
+ )
+
+ return (
+ func.call([input], _computed_options, memory_pool)
+ )
+
+
+def ceil(x, *, memory_pool=None):
+ """Round up to the nearest integer.
+
+ Compute the smallest integer value not less in magnitude than `x`.
+
+ This wraps the "ceil" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ceil')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'ceil',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def ceil_temporal(timestamps, multiple=1, unit='day', *,
+ week_starts_monday=True,
+ ceil_is_strictly_greater=False,
+ calendar_based_origin=False, options=None,
+ memory_pool=None):
+ """Round temporal values up to nearest multiple of specified time unit.
+
+ Null values emit null.
+ An error is returned if the values have a defined timezone but it
+ cannot be found in the timezone database.
+
+ This wraps the "ceil_temporal" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ timestamps : Array-like or scalar-like
+ Argument to compute function.
+ multiple : int, default 1
+ Number of units to round to.
+ unit : str, default "day"
+ The unit in which `multiple` is expressed.
+ Accepted values are "year", "quarter", "month", "week", "day",
+ "hour", "minute", "second", "millisecond", "microsecond",
+ "nanosecond".
+ week_starts_monday : bool, default True
+ If True, weeks start on Monday; if False, on Sunday.
+ ceil_is_strictly_greater : bool, default False
+ If True, ceil returns a rounded value that is strictly greater than
+ the
+ input. For example: ceiling 1970-01-01T00:00:00 to 3 hours would
+ yield 1970-01-01T03:00:00 if set to True and 1970-01-01T00:00:00
+ if set to False.
+ This applies to the ceil_temporal function only.
+ calendar_based_origin : bool, default False
+ By default, the origin is 1970-01-01T00:00:00. By setting this to
+ True,
+ rounding origin will be beginning of one less precise calendar unit.
+ E.g.: rounding to hours will use beginning of day as origin.
+
+ By default time is rounded to a multiple of units since
+ 1970-01-01T00:00:00. By setting calendar_based_origin to true,
+ time will be rounded to number of units since the last greater
+ calendar unit.
+ For example: rounding to multiple of days since the beginning of the
+ month or to hours since the beginning of the day.
+ Exceptions: week and quarter are not used as greater units,
+ therefore days will be rounded to the beginning of the month not
+ week. Greater unit of week is a year.
+ Note that ceiling and rounding might change sorting order of an array
+ near greater unit change. For example rounding YYYY-mm-dd 23:00:00 to
+ 5 hours will ceil and round to YYYY-mm-dd+1 01:00:00 and floor to
+ YYYY-mm-dd 20:00:00. On the other hand YYYY-mm-dd+1 00:00:00 will
+ ceil, round and floor to YYYY-mm-dd+1 00:00:00. This can break the
+ order of an already ordered array.
+ options : pyarrow.compute.RoundTemporalOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ceil_temporal',
+ pyarrow._compute.RoundTemporalOptions,
+ options,
+ (),
+ multiple=multiple,
+ unit=unit,
+ week_starts_monday=week_starts_monday,
+ ceil_is_strictly_greater=ceil_is_strictly_greater,
+ calendar_based_origin=calendar_based_origin
+ )
+ func = pyarrow._compute.get_function('ceil_temporal')
+
+ if isinstance(timestamps, Expression):
+ return Expression._call(
+ 'ceil_temporal',
+ [timestamps],
+ _computed_options
+ )
+
+ return (
+ func.call([timestamps], _computed_options, memory_pool)
+ )
+
+
+def choose(indices, *values, memory_pool=None):
+ """Choose values from several arrays.
+
+ For each row, the value of the first argument is used as a 0-based index
+ into the list of `values` arrays (i.e. index 0 selects the first of the
+ `values` arrays). The output value is the corresponding value of the
+ selected argument.
+
+ If an index is null, the output will be null.
+
+ This wraps the "choose" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ indices : Array-like or scalar-like
+ Argument to compute function.
+ *values : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('choose')
+ return (
+ func.call([indices, *values], memory_pool=memory_pool)
+ )
+
+
+def coalesce(*values, memory_pool=None):
+ """Select the first non-null value.
+
+ Each row of the output will be the value from the first corresponding
+ input
+ for which the value is not null. If all inputs are null in a row, the
+ output
+ will be null.
Review Comment:
Ah - I see where this is coming from. This is because the cpp-derived doc [for coalesce, as linked](https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/scalar_if_else.cc#L2739) has newlines which would be removed in the restructured text spec, but not before textwrap is being called. Let me work on this
From the cpp source referenced:
```cpp
const FunctionDoc coalesce_doc{
"Select the first non-null value",
("Each row of the output will be the value from the first corresponding input\n"
"for which the value is not null. If all inputs are null in a row, the output\n"
"will be null."),
{"*values"}};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- closed pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by "amol- (via GitHub)" <gi...@apache.org>.
amol- closed pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
URL: https://github.com/apache/arrow/pull/13126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1193349631
Hi, @jorisvandenbossche and @pitrou,
I've pushed new updates to this branch based on the comments, including the fully generated source files in `pyarrow/_compute_generated.py`
I've resolved updates to the tests and included the new compute functions that have been added since the original creation of the branch.
In line with the movement towards docutils, the following will test all of the examples that get pulled into the pyarrow.compute function documentation: `python -m doctest -v docs/additions/compute/*`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1177693668
@krcrouse thanks a lot for your work on this!
I didn't yet look in detail, but some general comments:
- I think we will probably want to commit the generated code to the repo (and in any case, doing that for now might also make it easier to review (although you already linked the file above as well)).
- Can you explain a bit more the need for `pyarrow.docutils`? (we should also make it a private module, or put in eg python/scripts or so?)
As far as I understand, by using the full ability of docutils, we can override _any_ section of the docstring (eg also overriding the `Parameters` section), instead of only _appending_ content to the docstring? (eg appending the example of notes section).
Do we think we will really need that full ability? Or it might in almost all cases be sufficient to just append content? (I think for all cases you currently have additions in this PR, appending would also be fine?) If appending is sufficient, that might keep the generation code simpler?
- > pyarrow.compute now imports from pyarrow.generated.compute
Small nit: this doesn't need to be a public module, so maybe something like `_compute_generated.py` instead?
- > Using raw reSt so that code block examples can be tested using doctest
We now started to tests docstring examples in general in the meantime (see https://github.com/apache/arrow/pull/13199, https://github.com/apache/arrow/pull/13216, https://github.com/apache/arrow/pull/13325), so we can probably test those docstrings that way as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1124470157
https://issues.apache.org/jira/browse/ARROW-12526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] krcrouse commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
krcrouse commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1124471692
To save you all from having to check out the branch and generate the code, attached is what the `python/pyarrow/generated/compute.py` file looks like at present (added as a .txt file because github rejects .py)
[compute.py.txt](https://github.com/apache/arrow/files/8674631/compute.py.txt)
.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1124470167
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on code in PR #13126:
URL: https://github.com/apache/arrow/pull/13126#discussion_r956029277
##########
python/docs/additions/compute/all.rst:
##########
@@ -0,0 +1,24 @@
+all
+===
+
+Examples
+--------
+
+.. code-block:: python
Review Comment:
For our other docstrings (in the code), we do this without the `code-block:: python` (which doctest supports, and sphinx as well to render). I suppose that can be done here as well?
See eg https://github.com/apache/arrow/blob/b832853ba62171d5fe5077681083fc6ea49bfd44/python/pyarrow/array.pxi#L180-L185
##########
python/docs/additions/compute/indices_nonzero.rst:
##########
@@ -0,0 +1,28 @@
+indices_nonzero
+===============
+
+Returns
+-------
+
+pyarrow.lib.UInt64Array
Review Comment:
```suggestion
pyarrow.UInt64Array
```
(we can use the public import name, in which case it will also automatically link)
##########
python/pyarrow/_rstutils.py:
##########
@@ -0,0 +1,291 @@
+# File GENERATED by scripts/generate_sources.py - DO NOT EDIT.
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+'''
+pyarrow._rstutils
Review Comment:
This module is only needed for the automatic generation, right? So in case we check in the generated file, this is only needed for developers, and can maybe be moved outside of pyarrow? (we don't need to ship this in the packages) For example also in python/scripts ?
##########
python/pyarrow/_compute_generated.py:
##########
@@ -0,0 +1,9381 @@
+# File GENERATED by scripts/generate_sources.py - DO NOT EDIT.
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyarrow
+import pyarrow._compute
+from pyarrow._compute import Expression
+
+
+def _handle_options(name, options_class, options, args, **kwargs):
+ if options is not None:
+ if isinstance(options, dict):
+ return options_class(**options)
+ elif isinstance(options, options_class):
+ return options
+ raise TypeError(
+ "Function {!r} expected a {} parameter, got {}"
+ .format(name, options_class, type(options)))
+
+ if args or kwargs:
+ # Note: This check is no longer permissable
+ # Generating function code with real signatures means that
+ # All of the keyword arguments have default values, and so
+ # this would always be true. As the default for the options object
+ # is always false, the options object takes precedence if provided.
+ #
+ # if options is not None:
+ # raise TypeError(
+ # "Function {!r} called with both an 'options' argument "
+ # "and additional arguments"
+ # .format(name))
+
+ return options_class(*args, **kwargs)
+
+ return None
+
+
+def abs(x, *, memory_pool=None):
+ """Calculate the absolute value of the argument element-wise.
+
+ Results will wrap around on integer overflow.
+ Use function "abs_checked" if you want overflow
+ to return an error.
+
+ This wraps the "abs" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('abs')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'abs',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def abs_checked(x, *, memory_pool=None):
+ """Calculate the absolute value of the argument element-wise.
+
+ This function returns an error on overflow. For a variant that
+ doesn't fail on overflow, use function "abs".
+
+ This wraps the "abs_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('abs_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'abs_checked',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def acos(x, *, memory_pool=None):
+ """Compute the inverse cosine.
+
+ NaN is returned for invalid input values;
+ to raise an error instead, see "acos_checked".
+
+ This wraps the "acos" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('acos')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'acos',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def acos_checked(x, *, memory_pool=None):
+ """Compute the inverse cosine.
+
+ Invalid input values raise an error;
+ to return NaN instead, see "acos".
+
+ This wraps the "acos_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('acos_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'acos_checked',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def add(x, y, *, memory_pool=None):
+ """Add the arguments element-wise.
+
+ Results will wrap around on integer overflow.
+ Use function "add_checked" if you want overflow
+ to return an error.
+
+ This wraps the "add" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('add')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'add',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def add_checked(x, y, *, memory_pool=None):
+ """Add the arguments element-wise.
+
+ This function returns an error on overflow. For a variant that
+ doesn't fail on overflow, use function "add".
+
+ This wraps the "add_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('add_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'add_checked',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def all(array, *, skip_nulls=True, min_count=1, options=None,
+ memory_pool=None):
+ """Test whether all elements in a boolean array evaluate to true.
+
+ Null values are ignored by default.
+ If the `skip_nulls` option is set to false, then Kleene logic is used.
+ See "kleene_and" for more details on Kleene logic.
+
+ This wraps the "all" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ skip_nulls : bool, default True
+ Whether to skip (ignore) nulls in the input.
+ If False, any null in the input forces the output to null.
+ min_count : int, default 1
+ Minimum number of non-null values in the input. If the number
+ of non-null values is below `min_count`, the output is null.
+ options : pyarrow.compute.ScalarAggregateOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+
+
+
+ Examples
+ --------
+ >>> import pyarrow as pa
+ >>> import pyarrow.compute as pc
+ >>> arr = pa.array([True, True, None, False, True])
+ >>> pc.all(arr)
+ <pyarrow.BooleanScalar: False>
+ >>> arr = pa.array([True, True, None, True, None, None])
+ >>> pc.all(arr)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.all(arr, skip_nulls = False)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.all(arr, min_count = 4)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.all(arr, min_count = 10)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.all(arr, min_count = 2)
+ <pyarrow.BooleanScalar: True>
+ """
+
+ _computed_options = _handle_options(
+ 'all',
+ pyarrow._compute.ScalarAggregateOptions,
+ options,
+ (),
+ skip_nulls=skip_nulls,
+ min_count=min_count
+ )
+ func = pyarrow._compute.get_function('all')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'all',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def and_(x, y, *, memory_pool=None):
+ """Logical 'and' boolean values.
+
+ When a null is encountered in either input, a null is output.
+ For a different null behavior, see function "and_kleene".
+
+ This wraps the "and" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def and_kleene(x, y, *, memory_pool=None):
+ """Logical 'and' boolean values (Kleene logic).
+
+ This function behaves as follows with nulls:
+
+ - true and null = null
+ - null and true = null
+ - false and null = false
+ - null and false = false
+ - null and null = null
+
+ In other words, in this context a null value really means "unknown",
+ and an unknown value 'and' false is always false.
+ For a different null behavior, see function "and".
+
+ This wraps the "and_kleene" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and_kleene')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_kleene',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def and_not(x, y, *, memory_pool=None):
+ """Logical 'and not' boolean values.
+
+ When a null is encountered in either input, a null is output.
+ For a different null behavior, see function "and_not_kleene".
+
+ This wraps the "and_not" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and_not')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_not',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def and_not_kleene(x, y, *, memory_pool=None):
+ """Logical 'and not' boolean values (Kleene logic).
+
+ This function behaves as follows with nulls:
+
+ - true and null = null
+ - null and false = null
+ - false and null = false
+ - null and true = false
+ - null and null = null
+
+ In other words, in this context a null value really means "unknown",
+ and an unknown value 'and not' true is always false, as is false
+ 'and not' an unknown value.
+ For a different null behavior, see function "and_not".
+
+ This wraps the "and_not_kleene" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('and_not_kleene')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'and_not_kleene',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def any(array, *, skip_nulls=True, min_count=1, options=None,
+ memory_pool=None):
+ """Test whether any element in a boolean array evaluates to true.
+
+ Null values are ignored by default.
+ If the `skip_nulls` option is set to false, then Kleene logic is used.
+ See "kleene_or" for more details on Kleene logic.
+
+ This wraps the "any" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ skip_nulls : bool, default True
+ Whether to skip (ignore) nulls in the input.
+ If False, any null in the input forces the output to null.
+ min_count : int, default 1
+ Minimum number of non-null values in the input. If the number
+ of non-null values is below `min_count`, the output is null.
+ options : pyarrow.compute.ScalarAggregateOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+
+
+
+ Examples
+ --------
+ >>> import pyarrow as pa
+ >>> import pyarrow.compute as pc
+ >>> arr = pa.array([True, True, None, False, True])
+ >>> pc.any(arr)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = 4)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = len(arr))
+ <pyarrow.BooleanScalar: None>
+ >>> arr = pa.array([False, False, None, False, True])
+ >>> pc.any(arr)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = 2)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any(arr, min_count = len(arr))
+ <pyarrow.BooleanScalar: None>
+ >>> pc.any(arr, skip_nulls = False)
+ <pyarrow.BooleanScalar: True>
+ >>> pc.any([False,None], skip_nulls = False)
+ <pyarrow.BooleanScalar: None>
+ >>> pc.any([False,None], skip_nulls = True)
+ <pyarrow.BooleanScalar: False>
+ """
+
+ _computed_options = _handle_options(
+ 'any',
+ pyarrow._compute.ScalarAggregateOptions,
+ options,
+ (),
+ skip_nulls=skip_nulls,
+ min_count=min_count
+ )
+ func = pyarrow._compute.get_function('any')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'any',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def approximate_median(array, *, skip_nulls=True, min_count=1, options=None,
+ memory_pool=None):
+ """Approximate median of a numeric array with T-Digest algorithm.
+
+ Nulls and NaNs are ignored.
+ A null scalar is returned if there is no valid data point.
+
+ This wraps the "approximate_median" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ skip_nulls : bool, default True
+ Whether to skip (ignore) nulls in the input.
+ If False, any null in the input forces the output to null.
+ min_count : int, default 1
+ Minimum number of non-null values in the input. If the number
+ of non-null values is below `min_count`, the output is null.
+ options : pyarrow.compute.ScalarAggregateOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'approximate_median',
+ pyarrow._compute.ScalarAggregateOptions,
+ options,
+ (),
+ skip_nulls=skip_nulls,
+ min_count=min_count
+ )
+ func = pyarrow._compute.get_function('approximate_median')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'approximate_median',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def array_filter(array, selection_filter, null_selection_behavior='drop', *,
+ options=None, memory_pool=None):
+ """Filter with a boolean selection filter.
+
+ The output is populated with values from the input `array` at positions
+ where the selection filter is non-zero. Nulls in the selection filter
+ are handled based on FilterOptions.
+
+ This wraps the "array_filter" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ selection_filter : Array-like
+ Argument to compute function.
+ null_selection_behavior : str, default "drop"
+ How to handle nulls in the selection filter.
+ Accepted values are "drop", "emit_null".
+ options : pyarrow.compute.FilterOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'array_filter',
+ pyarrow._compute.FilterOptions,
+ options,
+ (),
+ null_selection_behavior=null_selection_behavior
+ )
+ func = pyarrow._compute.get_function('array_filter')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'array_filter',
+ [array, selection_filter],
+ _computed_options
+ )
+
+ return (
+ func.call([array, selection_filter], _computed_options, memory_pool)
+ )
+
+
+def array_sort_indices(array, order='ascending', *, null_placement='at_end',
+ options=None, memory_pool=None):
+ """Return the indices that would sort an array.
+
+ This function computes an array of indices that define a stable sort
+ of the input array. By default, Null values are considered greater
+ than any other value and are therefore sorted at the end of the array.
+ For floating-point types, NaNs are considered greater than any
+ other non-null value, but smaller than null values.
+
+ The handling of nulls and NaNs can be changed in ArraySortOptions.
+
+ This wraps the "array_sort_indices" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ order : str, default "ascending"
+ Which order to sort values in.
+ Accepted values are "ascending", "descending".
+ null_placement : str, default "at_end"
+ Where nulls in the input should be sorted.
+ Accepted values are "at_start", "at_end".
+ options : pyarrow.compute.ArraySortOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'array_sort_indices',
+ pyarrow._compute.ArraySortOptions,
+ options,
+ (),
+ order=order,
+ null_placement=null_placement
+ )
+ func = pyarrow._compute.get_function('array_sort_indices')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'array_sort_indices',
+ [array],
+ _computed_options
+ )
+
+ return (
+ func.call([array], _computed_options, memory_pool)
+ )
+
+
+def array_take(array, indices, *, boundscheck=True, options=None,
+ memory_pool=None):
+ """Select values from an array based on indices from another array.
+
+ The output is populated with values from the input array at positions
+ given by `indices`. Nulls in `indices` emit null in the output.
+
+ This wraps the "array_take" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ array : Array-like
+ Argument to compute function.
+ indices : Array-like
+ Argument to compute function.
+ boundscheck : boolean, default True
+ Whether to check indices are within bounds. If False and an
+ index is out of boundes, behavior is undefined (the process
+ may crash).
+ options : pyarrow.compute.TakeOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'array_take',
+ pyarrow._compute.TakeOptions,
+ options,
+ (),
+ boundscheck=boundscheck
+ )
+ func = pyarrow._compute.get_function('array_take')
+
+ if isinstance(array, Expression):
+ return Expression._call(
+ 'array_take',
+ [array, indices],
+ _computed_options
+ )
+
+ return (
+ func.call([array, indices], _computed_options, memory_pool)
+ )
+
+
+def ascii_capitalize(strings, *, memory_pool=None):
+ """Capitalize the first character of ASCII input.
+
+ For each string in `strings`, return a capitalized version.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_capitalize" instead.
+
+ This wraps the "ascii_capitalize" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_capitalize')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_capitalize',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_center(strings, width=None, padding=' ', *, options=None,
+ memory_pool=None):
+ """Center strings by padding with a given character.
+
+ For each string in `strings`, emit a centered string by padding both
+ sides
+ with the given ASCII character.
+ Null values emit null.
+
+ This wraps the "ascii_center" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ width : int
+ Desired string length.
+ padding : str, default " "
+ What to pad the string with. Should be one byte or codepoint.
+ options : pyarrow.compute.PadOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_center',
+ pyarrow._compute.PadOptions,
+ options,
+ (),
+ width=width,
+ padding=padding
+ )
+ func = pyarrow._compute.get_function('ascii_center')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_center',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_is_alnum(strings, *, memory_pool=None):
+ """Classify strings as ASCII alphanumeric.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of alphanumeric ASCII characters. Null strings emit
+ null.
+
+ This wraps the "ascii_is_alnum" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_alnum')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_alnum',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_alpha(strings, *, memory_pool=None):
+ """Classify strings as ASCII alphabetic.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of alphabetic ASCII characters. Null strings emit
+ null.
+
+ This wraps the "ascii_is_alpha" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_alpha')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_alpha',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_decimal(strings, *, memory_pool=None):
+ """Classify strings as ASCII decimal.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of decimal ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_decimal" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_decimal')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_decimal',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_lower(strings, *, memory_pool=None):
+ """Classify strings as ASCII lowercase.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of lowercase ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_lower" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_lower')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_lower',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_printable(strings, *, memory_pool=None):
+ """Classify strings as ASCII printable.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of printable ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_printable" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_printable')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_printable',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_space(strings, *, memory_pool=None):
+ """Classify strings as ASCII whitespace.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of whitespace ASCII characters. Null strings emit
+ null.
+
+ This wraps the "ascii_is_space" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_space')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_space',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_title(strings, *, memory_pool=None):
+ """Classify strings as ASCII titlecase.
+
+ For each string in `strings`, emit true iff the string is title-cased,
+ i.e. it has at least one cased character, each uppercase character
+ follows an uncased character, and each lowercase character follows
+ an uppercase character.
+
+ This wraps the "ascii_is_title" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_title')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_title',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_is_upper(strings, *, memory_pool=None):
+ """Classify strings as ASCII uppercase.
+
+ For each string in `strings`, emit true iff the string is non-empty
+ and consists only of uppercase ASCII characters. Null strings emit null.
+
+ This wraps the "ascii_is_upper" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_is_upper')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_is_upper',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_lower(strings, *, memory_pool=None):
+ """Transform ASCII input to lowercase.
+
+ For each string in `strings`, return a lowercase version.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_lower" instead.
+
+ This wraps the "ascii_lower" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_lower')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_lower',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_lpad(strings, width=None, padding=' ', *, options=None,
+ memory_pool=None):
+ """Right-align strings by padding with a given character.
+
+ For each string in `strings`, emit a right-aligned string by prepending
+ the given ASCII character.
+ Null values emit null.
+
+ This wraps the "ascii_lpad" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ width : int
+ Desired string length.
+ padding : str, default " "
+ What to pad the string with. Should be one byte or codepoint.
+ options : pyarrow.compute.PadOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_lpad',
+ pyarrow._compute.PadOptions,
+ options,
+ (),
+ width=width,
+ padding=padding
+ )
+ func = pyarrow._compute.get_function('ascii_lpad')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_lpad',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_ltrim(strings, characters=None, *, options=None, memory_pool=None):
+ """Trim leading characters.
+
+ For each string in `strings`, remove any leading characters
+ from the `characters` option (as given in TrimOptions).
+ Null values emit null.
+ Both the `strings` and the `characters` are interpreted as
+ ASCII; to trim non-ASCII characters, use `utf8_ltrim`.
+
+ This wraps the "ascii_ltrim" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ characters : str
+ Individual characters to be trimmed from the string.
+ options : pyarrow.compute.TrimOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_ltrim',
+ pyarrow._compute.TrimOptions,
+ options,
+ (),
+ characters=characters
+ )
+ func = pyarrow._compute.get_function('ascii_ltrim')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_ltrim',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_ltrim_whitespace(strings, *, memory_pool=None):
+ """Trim leading ASCII whitespace characters.
+
+ For each string in `strings`, emit a string with leading ASCII whitespace
+ characters removed. Use `utf8_ltrim_whitespace` to trim leading Unicode
+ whitespace characters. Null values emit null.
+
+ This wraps the "ascii_ltrim_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_ltrim_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_ltrim_whitespace',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_reverse(strings, *, memory_pool=None):
+ """Reverse ASCII input.
+
+ For each ASCII string in `strings`, return a reversed version.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_reverse" instead.
+
+ This wraps the "ascii_reverse" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_reverse')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_reverse',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_rpad(strings, width=None, padding=' ', *, options=None,
+ memory_pool=None):
+ """Left-align strings by padding with a given character.
+
+ For each string in `strings`, emit a left-aligned string by appending
+ the given ASCII character.
+ Null values emit null.
+
+ This wraps the "ascii_rpad" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ width : int
+ Desired string length.
+ padding : str, default " "
+ What to pad the string with. Should be one byte or codepoint.
+ options : pyarrow.compute.PadOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_rpad',
+ pyarrow._compute.PadOptions,
+ options,
+ (),
+ width=width,
+ padding=padding
+ )
+ func = pyarrow._compute.get_function('ascii_rpad')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_rpad',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_rtrim(strings, characters=None, *, options=None, memory_pool=None):
+ """Trim trailing characters.
+
+ For each string in `strings`, remove any trailing characters
+ from the `characters` option (as given in TrimOptions).
+ Null values emit null.
+ Both the `strings` and the `characters` are interpreted as
+ ASCII; to trim non-ASCII characters, use `utf8_rtrim`.
+
+ This wraps the "ascii_rtrim" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ characters : str
+ Individual characters to be trimmed from the string.
+ options : pyarrow.compute.TrimOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_rtrim',
+ pyarrow._compute.TrimOptions,
+ options,
+ (),
+ characters=characters
+ )
+ func = pyarrow._compute.get_function('ascii_rtrim')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_rtrim',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_rtrim_whitespace(strings, *, memory_pool=None):
+ """Trim trailing ASCII whitespace characters.
+
+ For each string in `strings`, emit a string with trailing ASCII
+ whitespace
+ characters removed. Use `utf8_rtrim_whitespace` to trim trailing Unicode
+ whitespace characters. Null values emit null.
+
+ This wraps the "ascii_rtrim_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_rtrim_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_rtrim_whitespace',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_split_whitespace(strings, *, max_splits=None, reverse=False,
+ options=None,
+ memory_pool=None):
+ """Split string according to any ASCII whitespace.
+
+ Split each string according any non-zero length sequence of ASCII
+ whitespace characters. The output for each string input is a list
+ of strings.
+
+ The maximum number of splits and direction of splitting
+ (forward, reverse) can optionally be defined in SplitOptions.
+
+ This wraps the "ascii_split_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ max_splits : int or None, default None
+ Maximum number of splits for each input value (unlimited if None).
+ reverse : bool, default False
+ Whether to start splitting from the end of each input value.
+ This only has an effect if `max_splits` is not None.
+ options : pyarrow.compute.SplitOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_split_whitespace',
+ pyarrow._compute.SplitOptions,
+ options,
+ (),
+ max_splits=max_splits,
+ reverse=reverse
+ )
+ func = pyarrow._compute.get_function('ascii_split_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_split_whitespace',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_swapcase(strings, *, memory_pool=None):
+ """Transform ASCII input by inverting casing.
+
+ For each string in `strings`, return a string with opposite casing.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_swapcase" instead.
+
+ This wraps the "ascii_swapcase" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_swapcase')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_swapcase',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_title(strings, *, memory_pool=None):
+ """Titlecase each word of ASCII input.
+
+ For each string in `strings`, return a titlecased version.
+ Each word in the output will start with an uppercase character and its
+ remaining characters will be lowercase.
+
+ This function assumes the input is fully ASCII. If it may contain
+ non-ASCII characters, use "utf8_title" instead.
+
+ This wraps the "ascii_title" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_title')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_title',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_trim(strings, characters=None, *, options=None, memory_pool=None):
+ """Trim leading and trailing characters.
+
+ For each string in `strings`, remove any leading or trailing characters
+ from the `characters` option (as given in TrimOptions).
+ Null values emit null.
+ Both the `strings` and the `characters` are interpreted as
+ ASCII; to trim non-ASCII characters, use `utf8_trim`.
+
+ This wraps the "ascii_trim" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ characters : str
+ Individual characters to be trimmed from the string.
+ options : pyarrow.compute.TrimOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ascii_trim',
+ pyarrow._compute.TrimOptions,
+ options,
+ (),
+ characters=characters
+ )
+ func = pyarrow._compute.get_function('ascii_trim')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_trim',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def ascii_trim_whitespace(strings, *, memory_pool=None):
+ """Trim leading and trailing ASCII whitespace characters.
+
+ For each string in `strings`, emit a string with leading and trailing
+ ASCII
+ whitespace characters removed. Use `utf8_trim_whitespace` to trim Unicode
+ whitespace characters. Null values emit null.
+
+ This wraps the "ascii_trim_whitespace" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_trim_whitespace')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_trim_whitespace',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def ascii_upper(strings, *, memory_pool=None):
+ """Transform ASCII input to uppercase.
+
+ For each string in `strings`, return an uppercase version.
+
+ This function assumes the input is fully ASCII. It it may contain
+ non-ASCII characters, use "utf8_upper" instead.
+
+ This wraps the "ascii_upper" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ascii_upper')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'ascii_upper',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def asin(x, *, memory_pool=None):
+ """Compute the inverse sine.
+
+ NaN is returned for invalid input values;
+ to raise an error instead, see "asin_checked".
+
+ This wraps the "asin" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('asin')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'asin',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def asin_checked(x, *, memory_pool=None):
+ """Compute the inverse sine.
+
+ Invalid input values raise an error;
+ to return NaN instead, see "asin".
+
+ This wraps the "asin_checked" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('asin_checked')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'asin_checked',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def assume_timezone(timestamps, timezone=None, *, ambiguous='raise',
+ nonexistent='raise', options=None,
+ memory_pool=None):
+ """Convert naive timestamp to timezone-aware timestamp.
+
+ Input timestamps are assumed to be relative to the timezone given in the
+ `timezone` option. They are converted to UTC-relative timestamps and
+ the output type has its timezone set to the value of the `timezone`
+ option. Null values emit null.
+ This function is meant to be used when an external system produces
+ "timezone-naive" timestamps which need to be converted to
+ "timezone-aware" timestamps. An error is returned if the timestamps
+ already have a defined timezone.
+
+ This wraps the "assume_timezone" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ timestamps : Array-like or scalar-like
+ Argument to compute function.
+ timezone : str
+ Timezone to assume for the input.
+ ambiguous : str, default "raise"
+ How to handle timestamps that are ambiguous in the assumed timezone.
+ Accepted values are "raise", "earliest", "latest".
+ nonexistent : str, default "raise"
+ How to handle timestamps that don't exist in the assumed timezone.
+ Accepted values are "raise", "earliest", "latest".
+ options : pyarrow.compute.AssumeTimezoneOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'assume_timezone',
+ pyarrow._compute.AssumeTimezoneOptions,
+ options,
+ (),
+ timezone=timezone,
+ ambiguous=ambiguous,
+ nonexistent=nonexistent
+ )
+ func = pyarrow._compute.get_function('assume_timezone')
+
+ if isinstance(timestamps, Expression):
+ return Expression._call(
+ 'assume_timezone',
+ [timestamps],
+ _computed_options
+ )
+
+ return (
+ func.call([timestamps], _computed_options, memory_pool)
+ )
+
+
+def atan(x, *, memory_pool=None):
+ """Compute the inverse tangent of x.
+
+ The return value is in the range [-pi/2, pi/2];
+ for a full return range [-pi, pi], see "atan2".
+
+ This wraps the "atan" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('atan')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'atan',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def atan2(y, x, *, memory_pool=None):
+ """Compute the inverse tangent of y/x.
+
+ The return value is in the range [-pi, pi].
+
+ This wraps the "atan2" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ y : Array-like or scalar-like
+ Argument to compute function.
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('atan2')
+
+ if isinstance(y, Expression):
+ return Expression._call(
+ 'atan2',
+ [y, x]
+ )
+
+ return (
+ func.call([y, x], memory_pool=memory_pool)
+ )
+
+
+def binary_join(strings, separator, *, memory_pool=None):
+ """Join a list of strings together with a separator.
+
+ Concatenate the strings in `list`. The `separator` is inserted
+ between each given string.
+ Any null input and any null `list` element emits a null output.
+
+ This wraps the "binary_join" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ separator : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_join')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_join',
+ [strings, separator]
+ )
+
+ return (
+ func.call([strings, separator], memory_pool=memory_pool)
+ )
+
+
+def binary_join_element_wise(*strings, null_handling='emit_null',
+ null_replacement='',
+ options=None,
+ memory_pool=None):
+ """Join string arguments together, with the last argument as separator.
+
+ Concatenate the `strings` except for the last one. The last argument
+ in `strings` is inserted between each given string.
+ Any null separator element emits a null output. Null elements either
+ emit a null (the default), are skipped, or replaced with a given string.
+
+ This wraps the "binary_join_element_wise" compute function in the Arrow
+ C++ library.
+
+ Parameters
+ ----------
+ *strings : Array-like or scalar-like
+ Argument to compute function.
+ null_handling : str, default "emit_null"
+ How to handle null values in the inputs.
+ Accepted values are "emit_null", "skip", "replace".
+ null_replacement : str, default ""
+ Replacement string to emit for null inputs if `null_handling`
+ is "replace".
+ options : pyarrow.compute.JoinOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'binary_join_element_wise',
+ pyarrow._compute.JoinOptions,
+ options,
+ (),
+ null_handling=null_handling,
+ null_replacement=null_replacement
+ )
+ func = pyarrow._compute.get_function('binary_join_element_wise')
+ return (
+ func.call([*strings], _computed_options, memory_pool)
+ )
+
+
+def binary_length(strings, *, memory_pool=None):
+ """Compute string lengths.
+
+ For each string in `strings`, emit its length of bytes.
+ Null values emit null.
+
+ This wraps the "binary_length" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_length')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_length',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def binary_repeat(strings, num_repeats, *, memory_pool=None):
+ """Repeat a binary string.
+
+ For each binary string in `strings`, return a replicated version.
+
+ This wraps the "binary_repeat" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ num_repeats : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_repeat')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_repeat',
+ [strings, num_repeats]
+ )
+
+ return (
+ func.call([strings, num_repeats], memory_pool=memory_pool)
+ )
+
+
+def binary_replace_slice(strings, start=None, stop=None, replacement=None, *,
+ options=None,
+ memory_pool=None):
+ """Replace a slice of a binary string.
+
+ For each string in `strings`, replace a slice of the string defined by
+ `start`
+ and `stop` indices with the given `replacement`. `start` is inclusive
+ and `stop` is exclusive, and both are measured in bytes.
+ Null values emit null.
+
+ This wraps the "binary_replace_slice" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ start : int
+ Index to start slicing at (inclusive).
+ stop : int
+ Index to stop slicing at (exclusive).
+ replacement : str
+ What to replace the slice with.
+ options : pyarrow.compute.ReplaceSliceOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'binary_replace_slice',
+ pyarrow._compute.ReplaceSliceOptions,
+ options,
+ (),
+ start=start,
+ stop=stop,
+ replacement=replacement
+ )
+ func = pyarrow._compute.get_function('binary_replace_slice')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_replace_slice',
+ [strings],
+ _computed_options
+ )
+
+ return (
+ func.call([strings], _computed_options, memory_pool)
+ )
+
+
+def binary_reverse(strings, *, memory_pool=None):
+ """Reverse binary input.
+
+ For each binary string in `strings`, return a reversed version.
+
+ This function reverses the binary data at a byte-level.
+
+ This wraps the "binary_reverse" compute function in the Arrow C++
+ library.
+
+ Parameters
+ ----------
+ strings : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('binary_reverse')
+
+ if isinstance(strings, Expression):
+ return Expression._call(
+ 'binary_reverse',
+ [strings]
+ )
+
+ return (
+ func.call([strings], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_and(x, y, *, memory_pool=None):
+ """Bit-wise AND the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_and" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_and')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_and',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_not(x, *, memory_pool=None):
+ """Bit-wise negate the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_not" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_not')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_not',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_or(x, y, *, memory_pool=None):
+ """Bit-wise OR the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_or" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_or')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_or',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def bit_wise_xor(x, y, *, memory_pool=None):
+ """Bit-wise XOR the arguments element-wise.
+
+ Null values return null.
+
+ This wraps the "bit_wise_xor" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ y : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('bit_wise_xor')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'bit_wise_xor',
+ [x, y]
+ )
+
+ return (
+ func.call([x, y], memory_pool=memory_pool)
+ )
+
+
+def case_when(cond, *cases, memory_pool=None):
+ """Choose values based on multiple conditions.
+
+ `cond` must be a struct of Boolean values. `cases` can be a mix
+ of scalar and array arguments (of any type, but all must be the
+ same type or castable to a common type), with either exactly one
+ datum per child of `cond`, or one more `cases` than children of
+ `cond` (in which case we have an "else" value).
+
+ Each row of the output will be the corresponding value of the
+ first datum in `cases` for which the corresponding child of `cond`
+ is true, or otherwise the "else" value (if given), or null.
+
+ Essentially, this implements a switch-case or if-else, if-else...
+ statement.
+
+ This wraps the "case_when" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ cond : Array-like or scalar-like
+ Argument to compute function.
+ *cases : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('case_when')
+ return (
+ func.call([cond, *cases], memory_pool=memory_pool)
+ )
+
+
+def cast(input, target_type=None, *, allow_int_overflow=None,
+ allow_time_truncate=None, allow_time_overflow=None,
+ allow_decimal_truncate=None, allow_float_truncate=None,
+ allow_invalid_utf8=None, options=None, memory_pool=None):
+ """Cast values to another data type.
+
+ Behavior when values wouldn't fit in the target type
+ can be controlled through CastOptions.
+
+ This wraps the "cast" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ input : Array-like or scalar-like
+ Argument to compute function.
+ target_type : DataType, optional
+ The PyArrow type to cast to.
+ allow_int_overflow : bool, default False
+ Whether integer overflow is allowed when casting.
+ allow_time_truncate : bool, default False
+ Whether time precision truncation is allowed when casting.
+ allow_time_overflow : bool, default False
+ Whether date/time range overflow is allowed when casting.
+ allow_decimal_truncate : bool, default False
+ Whether decimal precision truncation is allowed when casting.
+ allow_float_truncate : bool, default False
+ Whether floating-point precision truncation is allowed when casting.
+ allow_invalid_utf8 : bool, default False
+ Whether producing invalid utf8 data is allowed when casting.
+ options : pyarrow.compute.CastOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'cast',
+ pyarrow._compute.CastOptions,
+ options,
+ (),
+ target_type=target_type,
+ allow_int_overflow=allow_int_overflow,
+ allow_time_truncate=allow_time_truncate,
+ allow_time_overflow=allow_time_overflow,
+ allow_decimal_truncate=allow_decimal_truncate,
+ allow_float_truncate=allow_float_truncate,
+ allow_invalid_utf8=allow_invalid_utf8
+ )
+ func = pyarrow._compute.get_function('cast')
+
+ if isinstance(input, Expression):
+ return Expression._call(
+ 'cast',
+ [input],
+ _computed_options
+ )
+
+ return (
+ func.call([input], _computed_options, memory_pool)
+ )
+
+
+def ceil(x, *, memory_pool=None):
+ """Round up to the nearest integer.
+
+ Compute the smallest integer value not less in magnitude than `x`.
+
+ This wraps the "ceil" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ x : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('ceil')
+
+ if isinstance(x, Expression):
+ return Expression._call(
+ 'ceil',
+ [x]
+ )
+
+ return (
+ func.call([x], memory_pool=memory_pool)
+ )
+
+
+def ceil_temporal(timestamps, multiple=1, unit='day', *,
+ week_starts_monday=True,
+ ceil_is_strictly_greater=False,
+ calendar_based_origin=False, options=None,
+ memory_pool=None):
+ """Round temporal values up to nearest multiple of specified time unit.
+
+ Null values emit null.
+ An error is returned if the values have a defined timezone but it
+ cannot be found in the timezone database.
+
+ This wraps the "ceil_temporal" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ timestamps : Array-like or scalar-like
+ Argument to compute function.
+ multiple : int, default 1
+ Number of units to round to.
+ unit : str, default "day"
+ The unit in which `multiple` is expressed.
+ Accepted values are "year", "quarter", "month", "week", "day",
+ "hour", "minute", "second", "millisecond", "microsecond",
+ "nanosecond".
+ week_starts_monday : bool, default True
+ If True, weeks start on Monday; if False, on Sunday.
+ ceil_is_strictly_greater : bool, default False
+ If True, ceil returns a rounded value that is strictly greater than
+ the
+ input. For example: ceiling 1970-01-01T00:00:00 to 3 hours would
+ yield 1970-01-01T03:00:00 if set to True and 1970-01-01T00:00:00
+ if set to False.
+ This applies to the ceil_temporal function only.
+ calendar_based_origin : bool, default False
+ By default, the origin is 1970-01-01T00:00:00. By setting this to
+ True,
+ rounding origin will be beginning of one less precise calendar unit.
+ E.g.: rounding to hours will use beginning of day as origin.
+
+ By default time is rounded to a multiple of units since
+ 1970-01-01T00:00:00. By setting calendar_based_origin to true,
+ time will be rounded to number of units since the last greater
+ calendar unit.
+ For example: rounding to multiple of days since the beginning of the
+ month or to hours since the beginning of the day.
+ Exceptions: week and quarter are not used as greater units,
+ therefore days will be rounded to the beginning of the month not
+ week. Greater unit of week is a year.
+ Note that ceiling and rounding might change sorting order of an array
+ near greater unit change. For example rounding YYYY-mm-dd 23:00:00 to
+ 5 hours will ceil and round to YYYY-mm-dd+1 01:00:00 and floor to
+ YYYY-mm-dd 20:00:00. On the other hand YYYY-mm-dd+1 00:00:00 will
+ ceil, round and floor to YYYY-mm-dd+1 00:00:00. This can break the
+ order of an already ordered array.
+ options : pyarrow.compute.RoundTemporalOptions, optional
+ Alternative way of passing options.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ _computed_options = _handle_options(
+ 'ceil_temporal',
+ pyarrow._compute.RoundTemporalOptions,
+ options,
+ (),
+ multiple=multiple,
+ unit=unit,
+ week_starts_monday=week_starts_monday,
+ ceil_is_strictly_greater=ceil_is_strictly_greater,
+ calendar_based_origin=calendar_based_origin
+ )
+ func = pyarrow._compute.get_function('ceil_temporal')
+
+ if isinstance(timestamps, Expression):
+ return Expression._call(
+ 'ceil_temporal',
+ [timestamps],
+ _computed_options
+ )
+
+ return (
+ func.call([timestamps], _computed_options, memory_pool)
+ )
+
+
+def choose(indices, *values, memory_pool=None):
+ """Choose values from several arrays.
+
+ For each row, the value of the first argument is used as a 0-based index
+ into the list of `values` arrays (i.e. index 0 selects the first of the
+ `values` arrays). The output value is the corresponding value of the
+ selected argument.
+
+ If an index is null, the output will be null.
+
+ This wraps the "choose" compute function in the Arrow C++ library.
+
+ Parameters
+ ----------
+ indices : Array-like or scalar-like
+ Argument to compute function.
+ *values : Array-like or scalar-like
+ Argument to compute function.
+ memory_pool : pyarrow.MemoryPool, optional
+ If not passed, will allocate memory from the default memory pool.
+ """
+
+ func = pyarrow._compute.get_function('choose')
+ return (
+ func.call([indices, *values], memory_pool=memory_pool)
+ )
+
+
+def coalesce(*values, memory_pool=None):
+ """Select the first non-null value.
+
+ Each row of the output will be the value from the first corresponding
+ input
+ for which the value is not null. If all inputs are null in a row, the
+ output
+ will be null.
Review Comment:
That's probably from trying to keep the line length within flake8 max? We might need to "reflow" the text then ... (or find a way to indicate to flake8 to ignore this file)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1177349457
@krcrouse Just a word to tell you that I hadn't a chance to take a look yet, but it's definitely on my TODO list (or may be switched to someone else's :-)). I hope the wait isn't too demotivating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13126: ARROW-12526: Pre-generating pyarrow.compute and creating a docstring additions system for pyarrow functions
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #13126:
URL: https://github.com/apache/arrow/pull/13126#issuecomment-1491436598
@krcrouse sorry for letting this slip my mind. I will try to take a look at the latest version and your last comment shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org