[GitHub] [arrow] westonpace commented on a diff in pull request #13709: Arrow-16000: [C++][Python] Dataset: Added transcoding function option to CSV scanner

[GitBox <gi...@apache.org>]

westonpace commented on code in PR #13709:
URL: https://github.com/apache/arrow/pull/13709#discussion_r931516618


##########
python/pyarrow/_dataset.pyx:
##########
@@ -1237,6 +1238,40 @@ cdef class CsvFileFormat(FileFormat):
         return f"<CsvFileFormat parse_options={self.parse_options}>"
 
 
+# From io.pxi
+def py_buffer(object obj):
+    """
+    Construct an Arrow buffer from a Python bytes-like or buffer-like object
+
+    Parameters
+    ----------
+    obj : object
+        the object from which the buffer should be constructed.
+    """
+    cdef shared_ptr[CBuffer] buf
+    buf = GetResultValue(PyBuffer.FromPyObject(obj))
+    return pyarrow_wrap_buffer(buf)
+
+
+# From io.pxi
+cdef void _cb_transform(transform_func, const shared_ptr[CBuffer]& src,
+                        shared_ptr[CBuffer]* dest) except *:
+    py_dest = transform_func(pyarrow_wrap_buffer(src))
+    dest[0] = pyarrow_unwrap_buffer(py_buffer(py_dest))
+
+
+# from io.pxi
+class Transcoder:

Review Comment:
   Hmm, do we really want to jump back to python here instead of using C++ utilities for decoding? (I'm not sure if there are any good standard utilities so maybe the answer is yes).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org