You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Daniel Nugent (Jira)" <ji...@apache.org> on 2020/06/18 22:30:00 UTC
[jira] [Commented] (ARROW-7702) [C++][Dataset] Provide (optional)
deterministic order of batches
[ https://issues.apache.org/jira/browse/ARROW-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140027#comment-17140027 ]
Daniel Nugent commented on ARROW-7702:
--------------------------------------
[~jorisvandenbossche] Please confirm that issue is now resolved.
> [C++][Dataset] Provide (optional) deterministic order of batches
> ----------------------------------------------------------------
>
> Key: ARROW-7702
> URL: https://issues.apache.org/jira/browse/ARROW-7702
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
> Reporter: Joris Van den Bossche
> Priority: Major
> Labels: dataset
>
> Example with python:
> {code}
> import pyarrow as pa
> import pyarrow.parquet as pq
> table = pa.table({'a': range(12)})
> pq.write_table(table, "test_chunks.parquet", chunk_size=3)
> # reading with dataset
> import pyarrow.dataset as ds
> ds.dataset("test_chunks.parquet").to_table().to_pandas()
> {code}
> gives non-deterministic result (order of the row groups in the parquet file):
> {code}
> In [25]: ds.dataset("test_chunks.parquet").to_table().to_pandas()
> Out[25]:
> a
> 0 0
> 1 1
> 2 2
> 3 3
> 4 4
> 5 5
> 6 6
> 7 7
> 8 8
> 9 9
> 10 10
> 11 11
> In [26]: ds.dataset("test_chunks.parquet").to_table().to_pandas()
> Out[26]:
> a
> 0 0
> 1 1
> 2 2
> 3 3
> 4 8
> 5 9
> 6 10
> 7 11
> 8 4
> 9 5
> 10 6
> 11 7
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)