You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "syun64 (via GitHub)" <gi...@apache.org> on 2024/06/02 02:31:08 UTC
[PR] to_arrow_batches to support returning Iterator[Recordbatch] instead of a fully materialized Arrow Table [iceberg-python]
syun64 opened a new pull request, #786:
URL: https://github.com/apache/iceberg-python/pull/786
- Iterator[RecordBatch] is preferable for memory constrained applications
- update `to_requested_schema` to take pa.RecordBatch instead of pa.Table
Question: pyarrow schema is currently inconsistent - it either returns pyarrow schema without Parquet metadata, or it adds the Parquet metadata and returns the value in some cases - we should make this consistent across our table scans and update the unit tests accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637416828
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
- arrow_table = pa.Table.from_batches(task.record_batches)
+
# if schema needs to be transformed, use the transformed schema and adjust the arrow table accordingly
# otherwise use the original schema
if (sanitized_schema := sanitize_column_names(table_schema)) != table_schema:
file_schema = sanitized_schema
- arrow_table = to_requested_schema(requested_schema=file_schema, file_schema=table_schema, table=arrow_table)
+ batches = [
+ to_requested_schema(requested_schema=file_schema, file_schema=table_schema, batch=batch)
+ for batch in task.record_batches
+ ]
else:
file_schema = table_schema
-
+ batches = task.record_batches
+ arrow_table = pa.Table.from_batches(batches)
Review Comment:
hmm, looking here, this forced materialization seems to preclude streaming writes, which would you may want if e.g. upserting large amounts of data. ParquetWriter can be used for streaming writes, so this seems unnecessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batches` to return Iterator[Recordbatch] instead of a fully materialized Arrow Table [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#issuecomment-2145268428
Thank you for sharing the additional context @djouallah
That's interesting! Although we don't use a pyarrow dataset in PyIceberg yet. We use a PyArrow dataset Fragment to create a dataset Scanner for each data file, and then directly create a PyArrow table instead of pushing down that layer into Arrow and using a PyArrow Dataset that represents the entire table of many files. I believe this is due to many outstanding issues that are being tracked together here on this issue: https://github.com/apache/iceberg-python/issues/30
My understanding of PyCapsuleInterface is that it is an interface for passing data back and forth Python and C layer if we wanted to write our own implementations of handling Arrow data format in the C layer. This is potentially an option, but I think the long-term roadmap for doing something more efficient for PyIceberg is to invest into Rust Python bindings as @Fokko mentioned at the Iceberg Summit.
In the interim, this PR aims to just add an extra API that returns an Iterator[RecordBatch] or a RecordBatchReader instead of a Table so that we don't have to fully materialize the data into memory when we are scanning for data.
Please let me know what you think about those above points, I want to make sure I'm understanding your suggestion correctly!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637386352
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1005,36 +1004,46 @@ def _task_to_table(
columns=[col.name for col in file_project_schema.columns],
)
- if positional_deletes:
- # Create the mask of indices that we're interested in
- indices = _combine_positional_deletes(positional_deletes, fragment.count_rows())
-
- if limit:
- if pyarrow_filter is not None:
- # In case of the filter, we don't exactly know how many rows
- # we need to fetch upfront, can be optimized in the future:
- # https://github.com/apache/arrow/issues/35301
- arrow_table = fragment_scanner.take(indices)
- arrow_table = arrow_table.filter(pyarrow_filter)
- arrow_table = arrow_table.slice(0, limit)
- else:
- arrow_table = fragment_scanner.take(indices[0:limit])
- else:
- arrow_table = fragment_scanner.take(indices)
+ current_index = 0
+ batches = fragment_scanner.to_batches()
Review Comment:
As I review this, it occurs to me that it might be useful to expose options related to batching/read ahead, etc, that pyarrow accepts when constructing the scanner. See the [pyarrow docs](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Scanner.html#pyarrow.dataset.Scanner.from_fragment) for more details.
Specifically, I think setting batch_size is probably something that ought to be tunable, since the memory pressure will be a function of batch size and the number and types of columns in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batches` to return Iterator[Recordbatch] instead of a fully materialized Arrow Table [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1624861038
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1005,36 +1004,42 @@ def _task_to_table(
columns=[col.name for col in file_project_schema.columns],
)
- if positional_deletes:
- # Create the mask of indices that we're interested in
- indices = _combine_positional_deletes(positional_deletes, fragment.count_rows())
-
- if limit:
- if pyarrow_filter is not None:
- # In case of the filter, we don't exactly know how many rows
- # we need to fetch upfront, can be optimized in the future:
- # https://github.com/apache/arrow/issues/35301
- arrow_table = fragment_scanner.take(indices)
- arrow_table = arrow_table.filter(pyarrow_filter)
- arrow_table = arrow_table.slice(0, limit)
- else:
- arrow_table = fragment_scanner.take(indices[0:limit])
- else:
- arrow_table = fragment_scanner.take(indices)
+ current_index = 0
+ batches = fragment_scanner.to_batches()
+ for batch in batches:
+ if positional_deletes:
+ # Create the mask of indices that we're interested in
+ indices = _combine_positional_deletes(positional_deletes, current_index, current_index + len(batch))
+ print(f"DEBUG: {indices=} {current_index=} {len(batch)=}")
+ print(f"{batch=}")
+ batch = batch.take(indices)
+ print(f"{batch=}")
# Apply the user filter
if pyarrow_filter is not None:
+ # we need to switch back and forth between RecordBatch and Table
+ # as Expression filter isn't yet supported in RecordBatch
+ # https://github.com/apache/arrow/issues/39220
+ arrow_table = pa.Table.from_batches([batch])
arrow_table = arrow_table.filter(pyarrow_filter)
- else:
- # If there are no deletes, we can just take the head
- # and the user-filter is already applied
- if limit:
- arrow_table = fragment_scanner.head(limit)
- else:
- arrow_table = fragment_scanner.to_table()
+ batch = arrow_table.to_batches()[0]
+ yield to_requested_schema(projected_schema, file_project_schema, batch)
+ current_index += len(batch)
- if len(arrow_table) < 1:
- return None
- return to_requested_schema(projected_schema, file_project_schema, arrow_table)
+
+def _task_to_table(
+ fs: FileSystem,
+ task: FileScanTask,
+ bound_row_filter: BooleanExpression,
+ projected_schema: Schema,
+ projected_field_ids: Set[int],
+ positional_deletes: Optional[List[ChunkedArray]],
+ case_sensitive: bool,
+ name_mapping: Optional[NameMapping] = None,
+) -> pa.Table:
+ batches = _task_to_record_batches(
+ fs, task, bound_row_filter, projected_schema, projected_field_ids, positional_deletes, case_sensitive, name_mapping
+ )
+ return pa.Table.from_batches(batches, schema=schema_to_pyarrow(projected_schema))
Review Comment:
This was exactly what I had in mind, looking good 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batches` to return Iterator[Recordbatch] instead of a fully materialized Arrow Table [iceberg-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#issuecomment-2144090634
sorry if I am confusing issue here, pyarrow dataset is awesome but still give substandard performance with Duckdb as statistics are not provided, particularly when doing joins reordering, my understanding PyCapsuleInterface fix that.
https://github.com/delta-io/delta-rs/issues/1838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637425908
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
- arrow_table = pa.Table.from_batches(task.record_batches)
+
# if schema needs to be transformed, use the transformed schema and adjust the arrow table accordingly
# otherwise use the original schema
if (sanitized_schema := sanitize_column_names(table_schema)) != table_schema:
file_schema = sanitized_schema
- arrow_table = to_requested_schema(requested_schema=file_schema, file_schema=table_schema, table=arrow_table)
+ batches = [
+ to_requested_schema(requested_schema=file_schema, file_schema=table_schema, batch=batch)
+ for batch in task.record_batches
+ ]
else:
file_schema = table_schema
-
+ batches = task.record_batches
+ arrow_table = pa.Table.from_batches(batches)
Review Comment:
Yep I totally agree. I wanted to focus this PR on introducing the reader first, and then work on a subsequent PR to incorporate batches into writes. This just maintains the existing functionality while making use of the refactored `to_requested_schema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637425676
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
- arrow_table = pa.Table.from_batches(task.record_batches)
+
# if schema needs to be transformed, use the transformed schema and adjust the arrow table accordingly
# otherwise use the original schema
if (sanitized_schema := sanitize_column_names(table_schema)) != table_schema:
file_schema = sanitized_schema
- arrow_table = to_requested_schema(requested_schema=file_schema, file_schema=table_schema, table=arrow_table)
+ batches = [
+ to_requested_schema(requested_schema=file_schema, file_schema=table_schema, batch=batch)
+ for batch in task.record_batches
+ ]
else:
file_schema = table_schema
-
+ batches = task.record_batches
+ arrow_table = pa.Table.from_batches(batches)
Review Comment:
i.e., maybe we could do something like the following?:
```python
def sanitize_batches(batches: Iterator[RecordBatch], table_schema: Schema, sanitized_schema: Schema) -> Iterator[RecordBatch]:
if sanitized_schema != table_schema:
for batch in batches:
yield to_requested_schema(requested_schema=sanitized_schema, file_schema=table_schema, batch=batch)
else:
yield from batches
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
# Check if schema needs to be transformed
sanitized_schema = sanitize_column_names(table_schema)
file_path = f'{table_metadata.location}/data/{task.generate_data_file_path("parquet")}'
fo = io.new_output(file_path)
with fo.create(overwrite=True) as fos:
with pq.ParquetWriter(fos, schema=sanitized_schema.as_arrow(), **parquet_writer_kwargs) as writer:
for sanitized_batch in sanitize_batches(task.record_batches, table_schema, sanitized_schema):
writer.write_table(pa.Table.from_batches([sanitized_batch]), row_group_size=row_group_size)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batches` to return Iterator[Recordbatch] instead of a fully materialized Arrow Table [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#issuecomment-2143893210
> how about PyCapsuleInterface https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
Hi @djouallah thank you for the suggestion, but I'm a bit confused here - what exactly did you have in mind for suggesting PyCapsule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637423967
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1005,36 +1004,46 @@ def _task_to_table(
columns=[col.name for col in file_project_schema.columns],
)
- if positional_deletes:
- # Create the mask of indices that we're interested in
- indices = _combine_positional_deletes(positional_deletes, fragment.count_rows())
-
- if limit:
- if pyarrow_filter is not None:
- # In case of the filter, we don't exactly know how many rows
- # we need to fetch upfront, can be optimized in the future:
- # https://github.com/apache/arrow/issues/35301
- arrow_table = fragment_scanner.take(indices)
- arrow_table = arrow_table.filter(pyarrow_filter)
- arrow_table = arrow_table.slice(0, limit)
- else:
- arrow_table = fragment_scanner.take(indices[0:limit])
- else:
- arrow_table = fragment_scanner.take(indices)
+ current_index = 0
+ batches = fragment_scanner.to_batches()
Review Comment:
That's a great suggestion @corleyma I'll adopt this feedback when I make the next round of changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637425676
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
- arrow_table = pa.Table.from_batches(task.record_batches)
+
# if schema needs to be transformed, use the transformed schema and adjust the arrow table accordingly
# otherwise use the original schema
if (sanitized_schema := sanitize_column_names(table_schema)) != table_schema:
file_schema = sanitized_schema
- arrow_table = to_requested_schema(requested_schema=file_schema, file_schema=table_schema, table=arrow_table)
+ batches = [
+ to_requested_schema(requested_schema=file_schema, file_schema=table_schema, batch=batch)
+ for batch in task.record_batches
+ ]
else:
file_schema = table_schema
-
+ batches = task.record_batches
+ arrow_table = pa.Table.from_batches(batches)
Review Comment:
i.e., maybe we could do something like the following?:
```python
# Check if schema needs to be transformed
sanitized_schema = sanitize_column_names(table_schema)
needs_sanitization = sanitized_schema != table_schema
file_path = f'{table_metadata.location}/data/{task.generate_data_file_path("parquet")}'
fo = io.new_output(file_path)
with fo.create(overwrite=True) as fos:
with pq.ParquetWriter(fos, schema=sanitized_schema.as_arrow(), **parquet_writer_kwargs) as writer:
for batch in task.record_batches:
if needs_sanitization:
sanitized_batch = to_requested_schema(
requested_schema=sanitized_schema, file_schema=table_schema, batch=batch
)
else:
sanitized_batch = batch
writer.write_table(pa.Table.from_batches([sanitized_batch]), row_group_size=row_group_size)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "corleyma (via GitHub)" <gi...@apache.org>.
corleyma commented on code in PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#discussion_r1637425676
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1795,15 +1873,19 @@ def write_file(io: FileIO, table_metadata: TableMetadata, tasks: Iterator[WriteT
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
- arrow_table = pa.Table.from_batches(task.record_batches)
+
# if schema needs to be transformed, use the transformed schema and adjust the arrow table accordingly
# otherwise use the original schema
if (sanitized_schema := sanitize_column_names(table_schema)) != table_schema:
file_schema = sanitized_schema
- arrow_table = to_requested_schema(requested_schema=file_schema, file_schema=table_schema, table=arrow_table)
+ batches = [
+ to_requested_schema(requested_schema=file_schema, file_schema=table_schema, batch=batch)
+ for batch in task.record_batches
+ ]
else:
file_schema = table_schema
-
+ batches = task.record_batches
+ arrow_table = pa.Table.from_batches(batches)
Review Comment:
i.e., maybe we could do something like the following?:
```python
def sanitize_batches(batches: Iterator[RecordBatch], table_schema: Schema, sanitized_schema: Schema) -> Iterator[RecordBatch]:
if sanitized_schema != table_schema:
for batch in batches:
yield to_requested_schema(requested_schema=sanitized_schema, file_schema=table_schema, batch=batch)
else:
yield from batches
def write_parquet(task: WriteTask) -> DataFile:
table_schema = task.schema
# Check if schema needs to be transformed
sanitized_schema = sanitize_column_names(table_schema)
file_path = f'{table_metadata.location}/data/{task.generate_data_file_path("parquet")}'
fo = io.new_output(file_path)
with fo.create(overwrite=True) as fos:
with pq.ParquetWriter(fos, schema=sanitized_schema.as_arrow(), **parquet_writer_kwargs) as writer:
for sanitized_batch in sanitized_batches_iterator(task.record_batches, table_schema, sanitized_schema):
writer.write_table(pa.Table.from_batches([sanitized_batch]), row_group_size=row_group_size)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batches` to return Iterator[Recordbatch] instead of a fully materialized Arrow Table [iceberg-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#issuecomment-2143798710
how about PyCapsuleInterface
https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Support `Table.to_arrow_batch_reader` to return RecordBatchReader instead of a fully materialized Arrow Table [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on PR #786:
URL: https://github.com/apache/iceberg-python/pull/786#issuecomment-2177312341
@HonahX @Fokko - could I ask your review on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org