You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2023/02/16 14:46:49 UTC
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #34099: GH-34098: [Python][Docs] Fix dataset docstring
jorisvandenbossche commented on code in PR #34099:
URL: https://github.com/apache/arrow/pull/34099#discussion_r1108573671
##########
python/pyarrow/_dataset.pyx:
##########
@@ -348,63 +479,297 @@ cdef class Dataset(_Weakrefable):
Parameters
----------
- **kwargs : dict, optional
- Arguments for `Scanner.from_dataset`.
+ columns : list of str, default None
+ The columns to project. This can be a list of column names to
+ include (order and duplicates will be preserved), or a dictionary
+ with {new_column_name: expression} values for more advanced
+ projections.
+
+ The list of columns or expressions may use the special fields
+ `__batch_index` (the index of the batch within the fragment),
+ `__fragment_index` (the index of the fragment within the dataset),
+ `__last_in_fragment` (whether the batch is last in fragment), and
+ `__filename` (the name of the source file or a description of the
+ source fragment).
+
+ The columns will be passed down to Datasets and corresponding data
+ fragments to avoid loading, copying, and deserializing columns
+ that will not be required further down the compute chain.
+ By default all of the available columns are projected. Raises
+ an exception if any of the referenced column names does not exist
+ in the dataset's Schema.
+ filter : Expression, default None
+ Scan will return only the rows matching the filter.
+ If possible the predicate will be pushed down to exploit the
+ partition information or internal metadata found in the data
+ source, e.g. Parquet statistics. Otherwise filters the loaded
+ RecordBatches before yielding them.
+ batch_size : int, default 128Ki
+ The maximum row count for scanned record batches. If scanned
+ record batches are overflowing memory then this method can be
+ called to reduce their size.
+ batch_readahead : int, default 16
+ The number of batches to read ahead in a file. This might not work
+ for all file formats. Increasing this number will increase
+ RAM usage but could also improve IO utilization.
+ fragment_readahead : int, default 4
+ The number of files to read ahead. Increasing this number will increase
+ RAM usage but could also improve IO utilization.
+ fragment_scan_options : FragmentScanOptions, default None
+ Options specific to a particular scan and fragment type, which
+ can change between different scans of the same dataset.
+ use_threads : bool, default True
+ If enabled, then maximum parallelism will be used determined by
+ the number of available CPU cores.
+ memory_pool : MemoryPool, default None
+ For memory allocations, if required. If not specified, uses the
+ default pool.
Returns
-------
table : Table
"""
- return self.scanner(**kwargs).to_table()
-
- def take(self, object indices, **kwargs):
+ return self.scanner(
+ columns=columns,
+ filter=filter,
+ batch_size=batch_size,
+ batch_readahead=batch_readahead,
+ fragment_readahead=fragment_readahead,
+ fragment_scan_options=fragment_scan_options,
+ use_threads=use_threads,
+ memory_pool=memory_pool
+ ).to_table()
+
+ def take(self,
+ object indices,
+ object columns=None,
+ Expression filter=None,
+ int batch_size=_DEFAULT_BATCH_SIZE,
+ int batch_readahead=_DEFAULT_BATCH_READAHEAD,
+ int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD,
+ FragmentScanOptions fragment_scan_options=None,
+ bint use_threads=True,
+ MemoryPool memory_pool=None):
"""
Select rows of data by index.
Parameters
----------
indices : Array or array-like
indices of rows to select in the dataset.
- **kwargs : dict, optional
- See scanner() method for full parameter description.
+ columns : list of str, default None
+ The columns to project. This can be a list of column names to
+ include (order and duplicates will be preserved), or a dictionary
+ with {new_column_name: expression} values for more advanced
+ projections.
+
+ The list of columns or expressions may use the special fields
+ `__batch_index` (the index of the batch within the fragment),
+ `__fragment_index` (the index of the fragment within the dataset),
+ `__last_in_fragment` (whether the batch is last in fragment), and
+ `__filename` (the name of the source file or a description of the
+ source fragment).
+
+ The columns will be passed down to Datasets and corresponding data
+ fragments to avoid loading, copying, and deserializing columns
+ that will not be required further down the compute chain.
+ By default all of the available columns are projected. Raises
+ an exception if any of the referenced column names does not exist
+ in the dataset's Schema.
+ filter : Expression, default None
+ Scan will return only the rows matching the filter.
+ If possible the predicate will be pushed down to exploit the
+ partition information or internal metadata found in the data
+ source, e.g. Parquet statistics. Otherwise filters the loaded
+ RecordBatches before yielding them.
+ batch_size : int, default 128Ki
+ The maximum row count for scanned record batches. If scanned
+ record batches are overflowing memory then this method can be
+ called to reduce their size.
+ batch_readahead : int, default 16
+ The number of batches to read ahead in a file. This might not work
+ for all file formats. Increasing this number will increase
+ RAM usage but could also improve IO utilization.
+ fragment_readahead : int, default 4
+ The number of files to read ahead. Increasing this number will increase
+ RAM usage but could also improve IO utilization.
+ fragment_scan_options : FragmentScanOptions, default None
+ Options specific to a particular scan and fragment type, which
+ can change between different scans of the same dataset.
+ use_threads : bool, default True
+ If enabled, then maximum parallelism will be used determined by
+ the number of available CPU cores.
+ memory_pool : MemoryPool, default None
+ For memory allocations, if required. If not specified, uses the
+ default pool.
Returns
-------
table : Table
"""
- return self.scanner(**kwargs).take(indices)
-
- def head(self, int num_rows, **kwargs):
+ return self.scanner(
+ columns=columns,
+ filter=filter,
+ batch_size=batch_size,
+ batch_readahead=batch_readahead,
+ fragment_readahead=fragment_readahead,
+ fragment_scan_options=fragment_scan_options,
+ use_threads=use_threads,
+ memory_pool=memory_pool
+ ).take(indices)
+
+ def head(self,
+ int num_rows,
+ object columns=None,
+ Expression filter=None,
+ int batch_size=_DEFAULT_BATCH_SIZE,
+ int batch_readahead=_DEFAULT_BATCH_READAHEAD,
+ int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD,
+ FragmentScanOptions fragment_scan_options=None,
+ bint use_threads=True,
+ MemoryPool memory_pool=None):
"""
Load the first N rows of the dataset.
Parameters
----------
num_rows : int
The number of rows to load.
- **kwargs : dict, optional
- See scanner() method for full parameter description.
+ columns : list of str, default None
+ The columns to project. This can be a list of column names to
+ include (order and duplicates will be preserved), or a dictionary
+ with {new_column_name: expression} values for more advanced
+ projections.
+
+ The list of columns or expressions may use the special fields
+ `__batch_index` (the index of the batch within the fragment),
+ `__fragment_index` (the index of the fragment within the dataset),
+ `__last_in_fragment` (whether the batch is last in fragment), and
+ `__filename` (the name of the source file or a description of the
+ source fragment).
+
+ The columns will be passed down to Datasets and corresponding data
+ fragments to avoid loading, copying, and deserializing columns
+ that will not be required further down the compute chain.
+ By default all of the available columns are projected. Raises
+ an exception if any of the referenced column names does not exist
+ in the dataset's Schema.
+ filter : Expression, default None
+ Scan will return only the rows matching the filter.
+ If possible the predicate will be pushed down to exploit the
+ partition information or internal metadata found in the data
+ source, e.g. Parquet statistics. Otherwise filters the loaded
+ RecordBatches before yielding them.
+ batch_size : int, default 128Ki
+ The maximum row count for scanned record batches. If scanned
+ record batches are overflowing memory then this method can be
+ called to reduce their size.
+ batch_readahead : int, default 16
+ The number of batches to read ahead in a file. This might not work
+ for all file formats. Increasing this number will increase
+ RAM usage but could also improve IO utilization.
+ fragment_readahead : int, default 4
+ The number of files to read ahead. Increasing this number will increase
+ RAM usage but could also improve IO utilization.
+ fragment_scan_options : FragmentScanOptions, default None
+ Options specific to a particular scan and fragment type, which
+ can change between different scans of the same dataset.
+ use_threads : bool, default True
+ If enabled, then maximum parallelism will be used determined by
+ the number of available CPU cores.
+ memory_pool : MemoryPool, default None
+ For memory allocations, if required. If not specified, uses the
+ default pool.
Returns
-------
table : Table
"""
- return self.scanner(**kwargs).head(num_rows)
-
- def count_rows(self, **kwargs):
+ return self.scanner(
+ columns=columns,
+ filter=filter,
+ batch_size=batch_size,
+ batch_readahead=batch_readahead,
+ fragment_readahead=fragment_readahead,
+ fragment_scan_options=fragment_scan_options,
+ use_threads=use_threads,
+ memory_pool=memory_pool
+ ).head(num_rows)
+
+ def count_rows(self,
+ object columns=None,
Review Comment:
For count_rows, doing a column selection doesn't really make sense? (it shouldn't influence the result)
Maybe we can keep it in `**kwargs` to be sure, but at least we can maybe limit its section in the docstring here
(starting to make special cases of course makes it harder to maintain the repeated docstrings ...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org