You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2023/06/19 05:15:33 UTC
[GitHub] [arrow] AlenkaF commented on a diff in pull request #36130: GH-36129: [Python] Consolidate common APIs in Table and RecordBatch
AlenkaF commented on code in PR #36130:
URL: https://github.com/apache/arrow/pull/36130#discussion_r1233503511
##########
python/pyarrow/table.pxi:
##########
@@ -1513,12 +1513,18 @@ cdef class _Tabular(_PandasConvertible):
return self.column(key)
+ def __len__(self):
+ return self.num_rows
+
def __repr__(self):
if not self._is_initialized():
raise ValueError("This object's internal pointer is NULL, do not "
"use any methods or attributes on this object")
return self.to_string(preview_cols=10)
+ def _column(self, int i):
+ raise NotImplementedError
+
Review Comment:
We plan to consolidate `_column` in another PR or we plan to keep it as is?
##########
python/pyarrow/table.pxi:
##########
@@ -1922,136 +2105,16 @@ cdef class RecordBatch(_Tabular):
self.batch = NULL
self._schema = None
- def __init__(self):
- raise TypeError("Do not call RecordBatch's constructor directly, use "
- "one of the `RecordBatch.from_*` functions instead.")
-
cdef void init(self, const shared_ptr[CRecordBatch]& batch):
self.sp_batch = batch
self.batch = batch.get()
def _is_initialized(self):
return self.batch != NULL
- @staticmethod
- def from_pydict(mapping, schema=None, metadata=None):
- """
- Construct a RecordBatch from Arrow arrays or columns.
-
- Parameters
- ----------
- mapping : dict or Mapping
- A mapping of strings to Arrays or Python lists.
- schema : Schema, default None
- If not passed, will be inferred from the Mapping values.
- metadata : dict or Mapping, default None
- Optional metadata for the schema (if inferred).
-
- Returns
- -------
- RecordBatch
-
- Examples
- --------
- >>> import pyarrow as pa
- >>> n_legs = [2, 2, 4, 4, 5, 100]
- >>> animals = ["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]
- >>> pydict = {'n_legs': n_legs, 'animals': animals}
-
- Construct a RecordBatch from arrays:
-
- >>> pa.RecordBatch.from_pydict(pydict)
- pyarrow.RecordBatch
- n_legs: int64
- animals: string
- ----
- n_legs: [2,2,4,4,5,100]
- animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
- >>> pa.RecordBatch.from_pydict(pydict).to_pandas()
- n_legs animals
- 0 2 Flamingo
- 1 2 Parrot
- 2 4 Dog
- 3 4 Horse
- 4 5 Brittle stars
- 5 100 Centipede
-
- Construct a RecordBatch with schema:
-
- >>> my_schema = pa.schema([
- ... pa.field('n_legs', pa.int64()),
- ... pa.field('animals', pa.string())],
- ... metadata={"n_legs": "Number of legs per animal"})
- >>> pa.RecordBatch.from_pydict(pydict, schema=my_schema).schema
- n_legs: int64
- animals: string
- -- schema metadata --
- n_legs: 'Number of legs per animal'
- """
-
- return _from_pydict(cls=RecordBatch,
- mapping=mapping,
- schema=schema,
- metadata=metadata)
-
- @staticmethod
- def from_pylist(mapping, schema=None, metadata=None):
- """
- Construct a RecordBatch from list of rows / dictionaries.
-
- Parameters
- ----------
- mapping : list of dicts of rows
- A mapping of strings to row values.
- schema : Schema, default None
- If not passed, will be inferred from the first row of the
- mapping values.
- metadata : dict or Mapping, default None
- Optional metadata for the schema (if inferred).
-
- Returns
- -------
- RecordBatch
-
- Examples
- --------
- >>> import pyarrow as pa
- >>> pylist = [{'n_legs': 2, 'animals': 'Flamingo'},
- ... {'n_legs': 4, 'animals': 'Dog'}]
- >>> pa.RecordBatch.from_pylist(pylist)
- pyarrow.RecordBatch
- n_legs: int64
- animals: string
- ----
- n_legs: [2,4]
- animals: ["Flamingo","Dog"]
-
- >>> pa.RecordBatch.from_pylist(pylist).to_pandas()
- n_legs animals
- 0 2 Flamingo
- 1 4 Dog
-
- Construct a RecordBatch with metadata:
-
- >>> my_metadata={"n_legs": "Number of legs per animal"}
- >>> pa.RecordBatch.from_pylist(pylist, metadata=my_metadata).schema
- n_legs: int64
- animals: string
- -- schema metadata --
- n_legs: 'Number of legs per animal'
- """
Review Comment:
And this part of an example.
##########
python/pyarrow/table.pxi:
##########
@@ -1787,6 +1941,35 @@ cdef class _Tabular(_PandasConvertible):
"""
return _pc().take(self, indices)
+ def to_pydict(self):
+ raise NotImplementedError
+
+ def to_pylist(self):
+ """
+ Convert the Table or RecordBatch to a list of rows / dictionaries.
+
+ Returns
+ -------
+ list
+
+ Examples
+ --------
+ Table (works similarly for RecordBatch)
+
+ >>> import pyarrow as pa
+ >>> import pandas as pd
+ >>> df = pd.DataFrame({'n_legs': [2, 4, 5, 100],
+ ... 'animals': ["Flamingo", "Horse", "Brittle stars", "Centipede"]})
+ >>> table = pa.Table.from_pandas(df)
+ >>> table.to_pylist()
+ [{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...
Review Comment:
This is actually a leftover from me, but now I think it is better to use `pa.table` or `pa.record_batch` instead of constructing a table from a pandas dataframe.
Can we do that here, similar to the example of `to_pylist` in the RecordBatch class?
##########
python/pyarrow/table.pxi:
##########
@@ -1922,136 +2105,16 @@ cdef class RecordBatch(_Tabular):
self.batch = NULL
self._schema = None
- def __init__(self):
- raise TypeError("Do not call RecordBatch's constructor directly, use "
- "one of the `RecordBatch.from_*` functions instead.")
-
cdef void init(self, const shared_ptr[CRecordBatch]& batch):
self.sp_batch = batch
self.batch = batch.get()
def _is_initialized(self):
return self.batch != NULL
- @staticmethod
- def from_pydict(mapping, schema=None, metadata=None):
- """
- Construct a RecordBatch from Arrow arrays or columns.
-
- Parameters
- ----------
- mapping : dict or Mapping
- A mapping of strings to Arrays or Python lists.
- schema : Schema, default None
- If not passed, will be inferred from the Mapping values.
- metadata : dict or Mapping, default None
- Optional metadata for the schema (if inferred).
-
- Returns
- -------
- RecordBatch
-
- Examples
- --------
- >>> import pyarrow as pa
- >>> n_legs = [2, 2, 4, 4, 5, 100]
- >>> animals = ["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]
- >>> pydict = {'n_legs': n_legs, 'animals': animals}
-
- Construct a RecordBatch from arrays:
-
- >>> pa.RecordBatch.from_pydict(pydict)
- pyarrow.RecordBatch
- n_legs: int64
- animals: string
- ----
- n_legs: [2,2,4,4,5,100]
- animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]
- >>> pa.RecordBatch.from_pydict(pydict).to_pandas()
- n_legs animals
- 0 2 Flamingo
- 1 2 Parrot
- 2 4 Dog
- 3 4 Horse
- 4 5 Brittle stars
- 5 100 Centipede
-
- Construct a RecordBatch with schema:
-
- >>> my_schema = pa.schema([
- ... pa.field('n_legs', pa.int64()),
- ... pa.field('animals', pa.string())],
- ... metadata={"n_legs": "Number of legs per animal"})
- >>> pa.RecordBatch.from_pydict(pydict, schema=my_schema).schema
- n_legs: int64
- animals: string
- -- schema metadata --
- n_legs: 'Number of legs per animal'
Review Comment:
Can we add this part of an example (schema) to the base class also?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org