You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/08/13 13:39:55 UTC

[GitHub] [arrow] bkietz commented on a change in pull request #7921: ARROW-9658: [Python] Python bindings for dataset writing

bkietz commented on a change in pull request #7921:
URL: https://github.com/apache/arrow/pull/7921#discussion_r469950141



##########
File path: python/pyarrow/_dataset.pyx
##########
@@ -2022,3 +2022,66 @@ def _get_partition_keys(Expression partition_expression):
         frombytes(name_val.first): pyarrow_wrap_scalar(name_val.second).as_py()
         for name_val in GetResultValue(CGetPartitionKeys(deref(expr.get())))
     }
+
+
+def _filesystemdataset_write(
+    data, object base_dir, Schema schema,
+    FileFormat format, FileSystem filesystem, Partitioning partitioning

Review comment:
       Should these be marked `not None`?

##########
File path: python/pyarrow/includes/libarrow_dataset.pxd
##########
@@ -126,6 +126,11 @@ cdef extern from "arrow/dataset/api.h" namespace "arrow::dataset" nogil:
     ctypedef CIterator[shared_ptr[CFragment]] CFragmentIterator \
         "arrow::dataset::FragmentIterator"
 
+    cdef cppclass CInMemoryFragment "arrow::dataset::InMemoryFragment"(
+            CFragment):
+        CInMemoryFragment(vector[shared_ptr[CRecordBatch]] record_batches,
+                          shared_ptr[CExpression] scalar)

Review comment:
       ```suggestion
                             shared_ptr[CExpression] partition_expression)
   ```

##########
File path: python/pyarrow/_dataset.pyx
##########
@@ -2022,3 +2022,66 @@ def _get_partition_keys(Expression partition_expression):
         frombytes(name_val.first): pyarrow_wrap_scalar(name_val.second).as_py()
         for name_val in GetResultValue(CGetPartitionKeys(deref(expr.get())))
     }
+
+
+def _filesystemdataset_write(
+    data, object base_dir, Schema schema,
+    FileFormat format, FileSystem filesystem, Partitioning partitioning
+):
+    """
+    CFileSystemDataset.Write wrapper
+    """
+    cdef:
+        c_string c_base_dir
+        shared_ptr[CSchema] c_schema
+        shared_ptr[CFileFormat] c_format
+        shared_ptr[CFileSystem] c_filesystem
+        shared_ptr[CPartitioning] c_partitioning
+        shared_ptr[CScanContext] c_context
+        # to create iterator of InMemory fragments
+        vector[shared_ptr[CRecordBatch]] c_batches
+        shared_ptr[CFragment] c_fragment
+        vector[shared_ptr[CFragment]] c_fragment_vector
+
+    c_base_dir = tobytes(_stringify_path(base_dir))
+    c_schema = pyarrow_unwrap_schema(schema)
+    c_format = format.unwrap()
+    c_filesystem = filesystem.unwrap()
+    c_partitioning = partitioning.unwrap()
+    # passthrough use_threads?
+    c_context = _build_scan_context()
+
+    if isinstance(data, Dataset):
+        with nogil:
+            check_status(
+                CFileSystemDataset.Write(
+                    c_schema,
+                    c_format,
+                    c_filesystem,
+                    c_base_dir,
+                    c_partitioning,
+                    c_context,
+                    (<Dataset> data).dataset.GetFragments()
+                )
+            )
+    else:
+        # data is list of batches
+        for batch in data:
+            c_batches.push_back((<RecordBatch> batch).sp_batch)
+
+        c_fragment = shared_ptr[CFragment](
+            new CInMemoryFragment(c_batches, _true.unwrap()))
+        c_fragment_vector.push_back(c_fragment)
+
+        with nogil:
+            check_status(
+                CFileSystemDataset.Write(
+                    c_schema,
+                    c_format,
+                    c_filesystem,
+                    c_base_dir,
+                    c_partitioning,
+                    c_context,
+                    MakeVectorIterator(c_fragment_vector)

Review comment:
       Currently we only parallelize over written fragments, so since this only creates a single written fragment use_threads is effectively ignored. I'd recommend creating a fragment wrapping each batch. This will result in one or more files per input record batch however.
    ```suggestion
           # data is list of batches
           for batch in data:
               c_batches.push_back((<RecordBatch> batch).sp_batch)
               c_fragment = shared_ptr[CFragment](
                   new CInMemoryFragment(c_batches, _true.unwrap()))
               c_batches.clear()
               c_fragment_vector.push_back(c_fragment)
   
           with nogil:
               check_status(
                   CFileSystemDataset.Write(
                       c_schema,
                       c_format,
                       c_filesystem,
                       c_base_dir,
                       c_partitioning,
                       c_context,
                       MakeVectorIterator(move(c_fragment_vector))
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org