You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/01/28 15:55:00 UTC
[jira] [Created] (ARROW-7702) [C++][Dataset] Provide (optional)
deterministic order of batches
Joris Van den Bossche created ARROW-7702:
--------------------------------------------
Summary: [C++][Dataset] Provide (optional) deterministic order of batches
Key: ARROW-7702
URL: https://issues.apache.org/jira/browse/ARROW-7702
Project: Apache Arrow
Issue Type: Bug
Components: C++ - Dataset, Python
Reporter: Joris Van den Bossche
Example with python:
{code}
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.table({'a': range(12)})
pq.write_table(table, "test_chunks.parquet", chunk_size=3)
# reading with dataset
import pyarrow.dataset as ds
ds.dataset("test_chunks.parquet").to_table().to_pandas()
{code}
gives non-deterministic result (order of the row groups in the parquet file):
```
In [25]: ds.dataset("test_chunks.parquet").to_table().to_pandas()
Out[25]:
a
0 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 10
11 11
In [26]: ds.dataset("test_chunks.parquet").to_table().to_pandas()
Out[26]:
a
0 0
1 1
2 2
3 3
4 8
5 9
6 10
7 11
8 4
9 5
10 6
11 7
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)