You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/19 02:41:02 UTC
[GitHub] [arrow] save-buffer opened a new pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
save-buffer opened a new pull request #11455:
URL: https://github.com/apache/arrow/pull/11455
Also adds a small test to make sure these methods do the right thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#discussion_r732820284
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,43 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch):
+ """
+ Write RecordBatch or Table to stream.
Review comment:
The "to stream", is that from a copy-paste from the RecordBatchWriter docs?
For writing a parquet file, that sounds a bit strange.
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,43 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch):
+ """
+ Write RecordBatch or Table to stream.
Review comment:
(and the same for `write_batch` / `write_table` below)
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,43 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch):
+ """
+ Write RecordBatch or Table to stream.
+
+ Parameters
+ ----------
+ table_or_batch : {RecordBatch, Table}
+ """
+ if isinstance(table_or_batch, pa.RecordBatch):
+ self.write_batch(table_or_batch)
+ elif isinstance(table_or_batch, pa.Table):
+ self.write_table(table_or_batch)
+ else:
+ raise ValueError(type(table_or_batch))
+
+ def write_batch(self, batch):
+ """
+ Write RecordBatch to stream.
+
+ Parameters
+ ----------
+ batch : RecordBatch
+ """
+ table = pa.Table.from_batches([batch], batch.schema)
+ self.write_table(table)
+
def write_table(self, table, row_group_size=None):
+ """
+ Write Table to stream in (contiguous) RecordBatch objects.
+
+ Parameters
+ ----------
+ table : Table
+ max_chunksize : int, default None
Review comment:
This is probably also from the RecordBatchWriter docs. For parquet, there is a `row_group_size` keyword that was not documented, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] save-buffer commented on a change in pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
save-buffer commented on a change in pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#discussion_r733947571
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,50 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch, row_group_size=None):
+ """
+ Write RecordBatch or Table to the Parquet file.
+
+ Parameters
+ ----------
+ table_or_batch : {RecordBatch, Table}
+ row_group_size : int, default None
+ Maximum size of each written row group. If None, the
+ row group size will be the same size as the input
+ table or batch.
Review comment:
Good point - I've clarified the doc string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] save-buffer commented on pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
save-buffer commented on pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#issuecomment-948896946
I've updated the docstring to clarify the maximum size, but please double check my wording
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#issuecomment-946318508
https://issues.apache.org/jira/browse/ARROW-13668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] save-buffer commented on pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
save-buffer commented on pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#issuecomment-947952582
Thanks for the feed back! I think I've addressed everything so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ElenaHenderson commented on pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
ElenaHenderson commented on pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#issuecomment-951822021
Benchmark runs are scheduled for baseline = 252e207495880eeff1f81e9fe28232b42bbd29ba and contender = 836ffa5656d5107fd4895ae8d7eb0e20a3df23ba. 836ffa5656d5107fd4895ae8d7eb0e20a3df23ba is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/e86e761d9d6c4481b36f3e9f5377358b...f709571206554b34836374c554aefc88/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/b05bac8898744dfc893c8e1372d78c53...f0cdb184226c42678b7c825a9fc7e72b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/ab2151bde9da46e09034fd90fa19e714...251443ac1d0e4357bb9cd720db758b15/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] save-buffer commented on a change in pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
save-buffer commented on a change in pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#discussion_r732940494
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,43 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch):
+ """
+ Write RecordBatch or Table to stream.
Review comment:
Ah yeah I just copy-pasted :) Should be better now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] save-buffer edited a comment on pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
save-buffer edited a comment on pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#issuecomment-947952582
Thanks for the feedback! I think I've addressed everything so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#discussion_r733394531
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,50 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch, row_group_size=None):
+ """
+ Write RecordBatch or Table to the Parquet file.
+
+ Parameters
+ ----------
+ table_or_batch : {RecordBatch, Table}
+ row_group_size : int, default None
+ Maximum size of each written row group. If None, the
+ row group size will be the same size as the input
+ table or batch.
Review comment:
There is actually a maximum size for the row group (if your table or recordbatch is really large):
https://github.com/apache/arrow/blob/542e81b6dea62f90817b117b1cb1b2de953f293e/cpp/src/parquet/properties.h#L97
Which corresponds to ~67 million rows. Not sure if this is worth mentioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on a change in pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
amol- commented on a change in pull request #11455:
URL: https://github.com/apache/arrow/pull/11455#discussion_r732633718
##########
File path: python/pyarrow/parquet.py
##########
@@ -687,7 +687,43 @@ def __exit__(self, *args, **kwargs):
# return false since we want to propagate exceptions
return False
+ def write(self, table_or_batch):
+ """
+ Write RecordBatch or Table to stream.
+
+ Parameters
+ ----------
+ table_or_batch : {RecordBatch, Table}
+ """
+ if isinstance(table_or_batch, pa.RecordBatch):
+ self.write_batch(table_or_batch)
+ elif isinstance(table_or_batch, pa.Table):
+ self.write_table(table_or_batch)
+ else:
+ raise ValueError(type(table_or_batch))
Review comment:
This should be a `TypeError`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace closed pull request #11455: ARROW-13668: [Python] Add `write_batch` and `write` methods to `ParquetWriter`
Posted by GitBox <gi...@apache.org>.
westonpace closed pull request #11455:
URL: https://github.com/apache/arrow/pull/11455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org