You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/05/24 16:45:47 UTC
[GitHub] [arrow] wesm opened a new pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
wesm opened a new pull request #7258:
URL: https://github.com/apache/arrow/pull/7258
This provides the `CAST(data AS target_type)` SQL idiom. The target_type is provided via CastOptions. As a result we no longer need to maintain separate binding boilerplate in Python for Array vs. ChunkedArray
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-634725333
Could someone review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-635505289
@jorisvandenbossche I can try moving it back, I was concerned about having to `cimport` things back into lib.pyx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-633258564
https://issues.apache.org/jira/browse/ARROW-8918
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-635372642
Is there a specific reason for moving compute into `pyarrow.lib` as well? (I would think multiple cython compilation units might be more "scalable" regarding code maintenance etc)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-634130309
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-635608038
Merging. The ARM failure (not sure what went wrong?) does not need to block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#discussion_r430274042
##########
File path: python/pyarrow/_compute.pyx
##########
@@ -316,3 +322,58 @@ cdef class FilterOptions(FunctionOptions):
cdef const CFunctionOptions* options(self) except NULL:
return &self.filter_options
+
+
+def cast(object arr, object target_type, bint safe=True):
+ """
+ Cast array values to another data type.
+
+ Example
+ -------
+
+ >>> from datetime import datetime
+ >>> import pyarrow as pa
+ >>> arr = pa.array([datetime(2010, 1, 1), datetime(2015, 1, 1)])
+ >>> arr.type
+ TimestampType(timestamp[us])
+
+ You can use ``pyarrow.DataType`` objects to specify the target type:
+
+ >>> arr.cast(pa.timestamp('ms'))
+ <pyarrow.lib.TimestampArray object at 0x10420eb88>
+ [
+ 1262304000000,
+ 1420070400000
Review comment:
Minor note, but for me (on master), this gives actual string date representation instead of milliseconds:
```
In [6]: arr.cast(pa.timestamp('ms'))
Out[6]:
<pyarrow.lib.TimestampArray object at 0x7f95c16bcc48>
[
2010-01-01 00:00:00.000,
2015-01-01 00:00:00.000
]
```
##########
File path: python/pyarrow/_compute.pyx
##########
@@ -316,3 +322,58 @@ cdef class FilterOptions(FunctionOptions):
cdef const CFunctionOptions* options(self) except NULL:
return &self.filter_options
+
+
+def cast(object arr, object target_type, bint safe=True):
+ """
+ Cast array values to another data type.
+
+ Example
+ -------
Review comment:
```suggestion
Examples
--------
```
(typically the examples are also put after Parameters / Returns sections)
##########
File path: python/pyarrow/_compute.pyx
##########
@@ -316,3 +322,58 @@ cdef class FilterOptions(FunctionOptions):
cdef const CFunctionOptions* options(self) except NULL:
return &self.filter_options
+
+
+def cast(object arr, object target_type, bint safe=True):
+ """
+ Cast array values to another data type.
+
+ Example
+ -------
+
+ >>> from datetime import datetime
+ >>> import pyarrow as pa
+ >>> arr = pa.array([datetime(2010, 1, 1), datetime(2015, 1, 1)])
+ >>> arr.type
+ TimestampType(timestamp[us])
+
+ You can use ``pyarrow.DataType`` objects to specify the target type:
+
+ >>> arr.cast(pa.timestamp('ms'))
+ <pyarrow.lib.TimestampArray object at 0x10420eb88>
+ [
+ 1262304000000,
+ 1420070400000
Review comment:
Ah, I see now it is just copied from the old docstrings below. That explains why the old repr is used here ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-635422952
For the rest looks good to me!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-635520799
What would need to be cimported in lib.pyx? I think in the Array and Table classes etc, for the computational methods, you do the `_pc.call_function(..)`. Or is it for the Option classes? (`_dataset.pyx` could also cimport that from `_compute`, like it does for the FileSystem for example)
(now, I don't know how important this is, it was just a throught)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm closed pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
wesm closed pull request #7258:
URL: https://github.com/apache/arrow/pull/7258
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7258: ARROW-8918: [C++][Python] Implement cast metafunction to allow use of "cast" with CallFunction, use in Python
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7258:
URL: https://github.com/apache/arrow/pull/7258#issuecomment-635541997
I added a _compute.pxd, no big deal. Will merge this once the build passes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org