You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2023/12/06 14:59:32 UTC
[PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
AlenkaF opened a new pull request, #39112:
URL: https://github.com/apache/arrow/pull/39112
### Rationale for this change
Legacy ParquetDataset has been deprecated for a while now, see https://github.com/apache/arrow/issues/31529. This PR is removing the legacy implementation from the code.
### What changes are included in this PR?
The PR is removing:
- `ParquetDatasetPiece `
- `ParquetManifest`
- `_ParquetDatasetMetadata `
- `ParquetDataset`
The PR is renaming `_ParquetDatasetV2` to `ParquetDataset` which was removed. It is also updating the docstrings.
The PR is updating:
- `read_table`
- `write_to_dataset`
The PR is updating all the tests to not use `use_legacy_dataset` keyword or legacy parametrisation.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Deprecated code is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1418573854
##########
docs/source/python/parquet.rst:
##########
@@ -511,36 +511,20 @@ from a remote filesystem into a pandas dataframe you may need to run
``sort_index`` to maintain row ordering (as long as the ``preserve_index``
option was enabled on write).
-.. note::
-
- The ParquetDataset is being reimplemented based on the new generic Dataset
- API (see the :ref:`dataset` docs for an overview). This is not yet the
- default, but can already be enabled by passing the ``use_legacy_dataset=False``
- keyword to :class:`ParquetDataset` or :func:`read_table`::
-
- pq.ParquetDataset('dataset_name/', use_legacy_dataset=False)
-
- Enabling this gives the following new features:
-
- - Filtering on all columns (using row group statistics) instead of only on
- the partition keys.
- - More fine-grained partitioning: support for a directory partitioning scheme
- in addition to the Hive-like partitioning (e.g. "/2019/11/15/" instead of
- "/year=2019/month=11/day=15/"), and the ability to specify a schema for
- the partition keys.
- - General performance improvement and bug fixes.
+Other features:
- It also has the following changes in behaviour:
+- Filtering on all columns (using row group statistics) instead of only on
+ the partition keys.
Review Comment:
Formatting nit: restructured text requires aligned indentation (same for below)
```suggestion
- Filtering on all columns (using row group statistics) instead of only on
the partition keys.
```
##########
python/pyarrow/parquet/core.py:
##########
@@ -3234,13 +2170,12 @@ def _mkdir_if_not_exists(fs, path):
def write_to_dataset(table, root_path, partition_cols=None,
partition_filename_cb=None, filesystem=None,
- use_legacy_dataset=None, schema=None,
Review Comment:
same here
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -63,21 +62,20 @@ def test_parquet_invalid_version(tempdir):
data_page_version="2.2")
-@parametrize_legacy_dataset
-def test_set_data_page_size(use_legacy_dataset):
+@pytest.mark.dataset
Review Comment:
I know this mark was in practice used through `@parametrize_legacy_dataset`, but we might not need to add it for all cases. Here, this `_check_roundtrip` is using `pq.read_table` under the hood. But for many cases (reading simple files), we have a fallback in `read_table` to use ParquetFile in case the dataset API is not available.
So I think we only need to mark those tests with `@pytest.mark.dataset` if it is actually testing a feature that is only supported in the dataset-based code path (eg reading multiple files instead of a single file, or when specifying a filter or partitioning)
##########
python/pyarrow/parquet/core.py:
##########
@@ -2420,18 +1380,80 @@ class _ParquetDatasetV2:
create a ParquetDataset object with filter:
>>> dataset = pq.ParquetDataset('dataset_v2/',
- ... filters=[('n_legs','=',4)],
- ... use_legacy_dataset=False)
+ ... filters=[('n_legs','=',4)])
>>> dataset.read().to_pandas()
n_legs animal year
0 4 Dog 2021
1 4 Horse 2022
- """
+"""
+
+
+class ParquetDataset:
+ __doc__ = """
+ Encapsulates details of reading a complete Parquet dataset possibly
+ consisting of multiple files and partitions in subdirectories.
+
+ Parameters
+ ----------
+ path_or_paths : str or List[str]
+ A directory name, single file name, or list of file names.
+ filesystem : FileSystem, default None
+ If nothing passed, will be inferred based on path.
+ Path will try to be found in the local on-disk filesystem otherwise
+ it will be parsed as an URI to determine the filesystem.
+ schema : pyarrow.parquet.Schema
+ Optionally provide the Schema for the Dataset, in which case it will
+ not be inferred from the source.
+ filters : pyarrow.compute.Expression or List[Tuple] or List[List[Tuple]],
+ default None
+ Rows which do not match the filter predicate will be removed from scanned
+ data. Partition keys embedded in a nested directory structure will be
+ exploited to avoid loading files at all if they contain no matching rows.
+ Within-file level filtering and different partitioning schemes are supported.
+
+ {1}
+ {0}
+ ignore_prefixes : list, optional
+ Files matching any of these prefixes will be ignored by the
+ discovery process.
+ This is matched to the basename of a path.
+ By default this is ['.', '_'].
+ Note that discovery happens only if a directory is passed as source.
+ pre_buffer : bool, default True
+ Coalesce and issue file reads in parallel to improve performance on
+ high-latency filesystems (e.g. S3, GCS). If True, Arrow will use a
+ background I/O thread pool. If using a filesystem layer that itself
+ performs readahead (e.g. fsspec's S3FS), disable readahead for best
+ results. Set to False if you want to prioritize minimal memory usage
+ over maximum speed.
+ coerce_int96_timestamp_unit : str, default None
+ Cast timestamps that are stored in INT96 format to a particular resolution
+ (e.g. 'ms'). Setting to None is equivalent to 'ns' and therefore INT96
+ timestamps will be inferred as timestamps in nanoseconds.
+ decryption_properties : FileDecryptionProperties or None
+ File-level decryption properties.
+ The decryption properties can be created using
+ ``CryptoFactory.file_decryption_properties()``.
+ thrift_string_size_limit : int, default None
+ If not None, override the maximum total string size allocated
+ when decoding Thrift structures. The default limit should be
+ sufficient for most Parquet files.
+ thrift_container_size_limit : int, default None
+ If not None, override the maximum total size of containers allocated
+ when decoding Thrift structures. The default limit should be
+ sufficient for most Parquet files.
+ page_checksum_verification : bool, default False
+ If True, verify the page checksum for each page read from the file.
+
+ Examples
+ --------
+ {2}
+ """.format(_read_docstring_common, _DNF_filter_doc, _parquet_dataset_example)
- def __init__(self, path_or_paths, filesystem=None, *, filters=None,
+ def __init__(self, path_or_paths, filesystem=None, schema=None, *, filters=None,
partitioning="hive", read_dictionary=None, buffer_size=None,
memory_map=False, ignore_prefixes=None, pre_buffer=True,
- coerce_int96_timestamp_unit=None, schema=None,
+ coerce_int96_timestamp_unit=None,
decryption_properties=None, thrift_string_size_limit=None,
thrift_container_size_limit=None,
page_checksum_verification=False,
Review Comment:
We might need to keep the `use_legacy_dataset` keyword here for a bit longer, but just ignore it / raise a deprecation warning when it is specified by the user
##########
python/pyarrow/parquet/core.py:
##########
@@ -2961,129 +1949,77 @@ def partitioning(self):
"""
-def read_table(source, *, columns=None, use_threads=True, metadata=None,
+def read_table(source, *, columns=None, use_threads=True,
schema=None, use_pandas_metadata=False, read_dictionary=None,
memory_map=False, buffer_size=0, partitioning="hive",
- filesystem=None, filters=None, use_legacy_dataset=False,
Review Comment:
same here about temporarily keeping the keyword around (eg if a user was now explicitly specifying `use_legacy_dataset=False` themselves in their code, that shouldn't break)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1434025854
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -986,31 +699,24 @@ def read_multiple_files(paths, columns=None, use_threads=True, **kwargs):
t = pa.Table.from_pandas(bad_apple)
_write_table(t, bad_apple_path)
- if not use_legacy_dataset:
- # TODO(dataset) Dataset API skips bad files
- return
+ # TODO(dataset) Dataset API skips bad files
Review Comment:
Yes, I think that's good, let's further figure this out in a follow-up issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1418889531
##########
python/pyarrow/parquet/core.py:
##########
@@ -2961,129 +1949,77 @@ def partitioning(self):
"""
-def read_table(source, *, columns=None, use_threads=True, metadata=None,
+def read_table(source, *, columns=None, use_threads=True,
schema=None, use_pandas_metadata=False, read_dictionary=None,
memory_map=False, buffer_size=0, partitioning="hive",
- filesystem=None, filters=None, use_legacy_dataset=False,
Review Comment:
Oh sorry, I meant to add this response to the first comment with the same subject: https://github.com/apache/arrow/pull/39112/files#r1418592766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1432655935
##########
python/pyarrow/tests/test_dataset.py:
##########
@@ -1388,16 +1380,13 @@ def _create_dataset_all_types(tempdir, chunk_size=None):
path = str(tempdir / "test_parquet_dataset_all_types")
# write_to_dataset currently requires pandas
Review Comment:
Oh yes, it was needed for `partition_cols` in the legacy path: https://github.com/apache/arrow/blob/87865b5a85c722ef7578aed4300e9d0b219c909c/python/pyarrow/parquet/core.py#L3516
Will remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866609381
Still need to look into the failures with `Unrecognized filesystem` error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1432648346
##########
python/pyarrow/tests/parquet/test_pandas.py:
##########
@@ -497,15 +447,14 @@ def test_categorical_index_survives_roundtrip(use_legacy_dataset):
table = pa.Table.from_pandas(df)
bos = pa.BufferOutputStream()
pq.write_table(table, bos)
- ref_df = pq.read_pandas(
- bos.getvalue(), use_legacy_dataset=use_legacy_dataset).to_pandas()
+ ref_df = pq.read_pandas(bos.getvalue()).to_pandas()
assert isinstance(ref_df.index, pd.CategoricalIndex)
assert ref_df.index.equals(df.index)
@pytest.mark.pandas
-@parametrize_legacy_dataset
-def test_categorical_order_survives_roundtrip(use_legacy_dataset):
+@pytest.mark.dataset
Review Comment:
I removed all dataset marks, looks like the tests are using `pq.write_table`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866614531
Ah, I guess the issue is the same it only fails in `_ensure_filesystem` and not in `read_parquet`. Will add xfail marks there also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866404553
Yeah, missed some metadata there. Thanks, will correct!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1850541037
It is very hard to review this PR due to the way the diff is presented in GitHub. I tried to summarise the main changes in the description of the PR, hope it helps a bit.
@jorisvandenbossche after the last review I have updated the marks in the tests, added `use_legacy_dataset=None` to `ParquetDataset` class, `read_table` and `arite_to_dataset`.
I have also removed `** kwargs` from `ParquetDataset`, previously `_ParquetDatasetV2`, which meant I had to remove the code connected to `metadata`, `split_row_groups` and `validate_schema` ([raising an error](https://github.com/apache/arrow/blob/92e56ba8906f40996cc81bc09fca10c4d53b32fa/python/pyarrow/parquet/core.py#L2441-L2448)) and so I added a note in the docstrings https://github.com/apache/arrow/pull/39112/commits/b6799cfbe68a3289e00378739bc5291ff81d8855. Not sure if that is well done though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1862417546
Revision: 0cbd03dcbafb76853c0eabb08d74795afd77cc4a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-39c3c2ced4](https://github.com/ursacomputing/crossbow/branches/all?query=actions-39c3c2ced4)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/7259919512/job/19778013093)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/7259919496/job/19778013100)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866177526
@github-actions crossbow submit example-python-minimal-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1418876351
##########
python/pyarrow/parquet/core.py:
##########
@@ -2961,129 +1949,77 @@ def partitioning(self):
"""
-def read_table(source, *, columns=None, use_threads=True, metadata=None,
+def read_table(source, *, columns=None, use_threads=True,
schema=None, use_pandas_metadata=False, read_dictionary=None,
memory_map=False, buffer_size=0, partitioning="hive",
- filesystem=None, filters=None, use_legacy_dataset=False,
Review Comment:
Specifying `use_legacy_dataset=False` doesn't break in the current code due to **kwargs kept in `ParquetDataset` from `_ParquetDatasetV2`. I would personally keep it as is as it gets ignored currently and would add a deprecation warning and a test for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1423777049
##########
python/pyarrow/parquet/core.py:
##########
@@ -1772,7 +1774,8 @@ def read_table(source, *, columns=None, use_threads=True,
if metadata is not None:
raise ValueError(
"The 'metadata' keyword is no longer supported with the new "
- "datasets-based implementation."
+ "datasets-based implementation and will be removed in future "
+ "versions."
Review Comment:
For `read_table` here, we can maybe also just remove that (since it was already raising before)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866476155
I am not sure if raising for the `metadata` will be sufficient? My understanding is that those tests (for the legacy HDFS filesystem) are using hdfs.read_parquet -> pq.read_table by passing a legacy HDFS filesystem object, and with removing the legacy parquet code, we no longer support the legacy filesystems (and now as a next step, we can actually also remove those legacy filesystems).
So maybe we can just "xfail" those tests for now? And then when removing the HDFS legacy filesystem, we can see if we just remove those tests, or if we rewrite them using the new hdfs filesystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866456900
@github-actions crossbow submit -g python-*-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866174507
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #39112:
URL: https://github.com/apache/arrow/pull/39112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866399401
The HDFS failures seem to be related: https://github.com/ursacomputing/crossbow/actions/runs/7288206604/job/19860362598#step:6:9473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866459400
```
Invalid group(s) {'python-*-hdfs'}. Must be one of {'integration', 'example-cpp', 'verify-rc-source', 'fuzz', 'nightly', 'python', 'nightly-release', 'verify-rc-binaries', 'cpp', 'c-glib', 'conan', 'java', 'linux', 'go', 'example', 'verify-rc-wheels', 'r', 'packaging', 'test', 'nightly-tests', 'example-python', 'vcpkg', 'verify-rc-jars', 'linux-amd64', 'conda', 'homebrew', 'ruby', 'linux-arm64', 'verify-rc', 'verify-rc-source-macos', 'wheel', 'verify-rc-source-linux', 'nightly-packaging'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7289911234
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866443883
@jorisvandenbossche if I am understanding correctly `read_parquet` from the failed build is part of a legacy `Filesystem` and will be removed? So I can for time being add a `ValueError` if `metadata` is specified or I can just remove it from the signature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1854199146
Revision: 0cbd03dcbafb76853c0eabb08d74795afd77cc4a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-650c03de3f](https://github.com/ursacomputing/crossbow/branches/all?query=actions-650c03de3f)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778263/job/19605694903)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777584/job/19605692836)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777834/job/19605693428)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778269/job/19605694906)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778014/job/19605694432)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777988/job/19605694218)|
|test-conda-python-3.10-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777381/job/19605691683)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778040/job/19605694506)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777779/job/19605693177)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777476/job/19605691960)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777518/job/19605692175)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777483/job/19605691852)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778305/job/19605694933)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777682/job/19605693114)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778081/job/19605694375)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777840/job/19605693521)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777472/job/19605692031)|
|test-conda-python-3.8-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7197778164/job/19605694924)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777708/job/19605693038)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777947/job/19605694114)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777558/job/19605692622)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/19605696775)|
|test-fedora-38-python-3|[](https://github.com/ursacomputing/crossbow/runs/19605696585)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/19605695902)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/7197777802/job/19605693277)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1418882389
##########
python/pyarrow/parquet/core.py:
##########
@@ -3234,13 +2170,12 @@ def _mkdir_if_not_exists(fs, path):
def write_to_dataset(table, root_path, partition_cols=None,
partition_filename_cb=None, filesystem=None,
- use_legacy_dataset=None, schema=None,
Review Comment:
I will add it back here as it gets caught by `write_options = parquet_format.make_write_options(**kwargs)` and raises an error:
```
>>> pq.write_to_dataset(table, root_path='dataset_v2',
... partition_cols=['year'],
... use_legacy_dataset=None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/alenkafrim/repos/arrow/python/pyarrow/parquet/core.py", line 2113, in write_to_dataset
write_options = parquet_format.make_write_options(**kwargs)
File "pyarrow/_dataset_parquet.pyx", line 201, in pyarrow._dataset_parquet.ParquetFileFormat.make_write_options
File "pyarrow/_dataset_parquet.pyx", line 578, in pyarrow._dataset_parquet.ParquetFileWriteOptions.update
TypeError: unexpected parquet write option: use_legacy_dataset
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866180190
Revision: 33bf4920ce21dbc4ed45bc7eefec97b62f4054c3
Submitted crossbow builds: [ursacomputing/crossbow @ actions-3193efe765](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3193efe765)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/7288231493/job/19860432268)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/7288231476/job/19860432263)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1433859749
##########
python/pyarrow/tests/parquet/test_pandas.py:
##########
@@ -150,52 +144,22 @@ def test_pandas_parquet_2_0_roundtrip_read_pandas_no_index_written(
tm.assert_frame_equal(df, df_read)
-# TODO(dataset) duplicate column selection actually gives duplicate columns now
Review Comment:
https://github.com/apache/arrow/pull/39112/commits/5b1f6eda7b8696989ff4487171770677912c58fb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866630233
Revision: 481a85c47ce45243ba972a3287b31675ac3ac76e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-fc72b69813](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fc72b69813)
|Task|Status|
|----|------|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/7290904011/job/19868708412)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/7290904092/job/19868708902)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1433851105
##########
python/pyarrow/tests/parquet/test_pandas.py:
##########
@@ -150,52 +144,22 @@ def test_pandas_parquet_2_0_roundtrip_read_pandas_no_index_written(
tm.assert_frame_equal(df, df_read)
-# TODO(dataset) duplicate column selection actually gives duplicate columns now
Review Comment:
I think there is only one similar test using datasets
https://github.com/apache/arrow/blob/3c66491846a24f17014b31a22fafdda0229f881a/python/pyarrow/tests/test_dataset.py#L3472-L3473
which is same at the end but will still add another for `pq.read_table` which is now using only the new dataset API. Not sure if I have to roundtrip with pandas though?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866177500
Revision: 33bf4920ce21dbc4ed45bc7eefec97b62f4054c3
Submitted crossbow builds: [ursacomputing/crossbow @ actions-878c4b2b4c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-878c4b2b4c)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206259/job/19860360664)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206606/job/19860362576)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206604/job/19860362598)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206535/job/19860362113)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206556/job/19860362109)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206404/job/19860361157)|
|test-conda-python-3.10-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206358/job/19860361147)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206470/job/19860361651)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206599/job/19860362568)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206545/job/19860362132)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206544/job/19860362120)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206298/job/19860360908)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206486/job/19860361645)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206565/job/19860362289)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206244/job/19860360705)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206245/job/19860360668)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206409/job/19860361209)|
|test-conda-python-3.8-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206602/job/19860362551)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206469/job/19860361646)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206301/job/19860360920)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206466/job/19860361406)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/19860362074)|
|test-fedora-38-python-3|[](https://github.com/ursacomputing/crossbow/runs/19860362095)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/19860361672)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/7288206614/job/19860362749)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1434024893
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -986,31 +699,24 @@ def read_multiple_files(paths, columns=None, use_threads=True, **kwargs):
t = pa.Table.from_pandas(bad_apple)
_write_table(t, bad_apple_path)
- if not use_legacy_dataset:
- # TODO(dataset) Dataset API skips bad files
- return
+ # TODO(dataset) Dataset API skips bad files
Review Comment:
Can also open an issue and resolve this TODO in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1431145654
##########
python/pyarrow/parquet/core.py:
##########
@@ -1714,11 +1185,13 @@ def _open_dataset_file(dataset, path, meta=None):
class ParquetDataset:
-
__doc__ = """
Encapsulates details of reading a complete Parquet dataset possibly
consisting of multiple files and partitions in subdirectories.
+Keywords `metadata`, `split_row_groups=True` and `validate_schema=False`
+are not yet supported.
Review Comment:
We could maybe also leave this out? The wording now suggests that we are planning to support them, and I am not sure if that is the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1433942636
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -986,31 +699,24 @@ def read_multiple_files(paths, columns=None, use_threads=True, **kwargs):
t = pa.Table.from_pandas(bad_apple)
_write_table(t, bad_apple_path)
- if not use_legacy_dataset:
- # TODO(dataset) Dataset API skips bad files
- return
+ # TODO(dataset) Dataset API skips bad files
Review Comment:
I haven't found one. I am trying to figure out what should be tested and will update it so that skipping bad files in Dataset API is included in it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1431112710
##########
python/pyarrow/tests/parquet/test_compliant_nested_type.py:
##########
@@ -83,21 +79,18 @@ def test_write_compliant_nested_type_enable(tempdir,
assert new_table.schema.types[0].value_field.name == 'element'
# Verify that the new table can be read/written correctly
- _check_roundtrip(new_table,
- use_legacy_dataset=use_legacy_dataset)
+ _check_roundtrip(new_table)
@pytest.mark.pandas
-@parametrize_legacy_dataset
+@pytest.mark.dataset
Review Comment:
Is dataset mark needed here?
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -752,17 +682,15 @@ def test_fastparquet_cross_compatibility(tempdir):
tm.assert_frame_equal(table_fp.to_pandas(), df)
-@parametrize_legacy_dataset
@pytest.mark.parametrize('array_factory', [
lambda: pa.array([0, None] * 10),
lambda: pa.array([0, None] * 10).dictionary_encode(),
lambda: pa.array(["", None] * 10),
lambda: pa.array(["", None] * 10).dictionary_encode(),
])
-@pytest.mark.parametrize('use_dictionary', [False, True])
Review Comment:
Is there a reason this parametrization was also removed?
##########
python/pyarrow/tests/test_dataset.py:
##########
@@ -1388,16 +1380,13 @@ def _create_dataset_all_types(tempdir, chunk_size=None):
path = str(tempdir / "test_parquet_dataset_all_types")
# write_to_dataset currently requires pandas
Review Comment:
This comment is no longer correct I think? (so that might also mean that all tests that use this helper no longer need the pandas mark)
##########
python/pyarrow/tests/parquet/test_datetime.py:
##########
@@ -48,8 +47,8 @@
@pytest.mark.pandas
-@parametrize_legacy_dataset
-def test_pandas_parquet_datetime_tz(use_legacy_dataset):
+@pytest.mark.dataset
Review Comment:
Same here and the one below about the dataset marker
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -986,31 +699,24 @@ def read_multiple_files(paths, columns=None, use_threads=True, **kwargs):
t = pa.Table.from_pandas(bad_apple)
_write_table(t, bad_apple_path)
- if not use_legacy_dataset:
- # TODO(dataset) Dataset API skips bad files
- return
+ # TODO(dataset) Dataset API skips bad files
Review Comment:
Do we know if there is an issue about this?
##########
python/pyarrow/tests/parquet/test_pandas.py:
##########
@@ -497,15 +447,14 @@ def test_categorical_index_survives_roundtrip(use_legacy_dataset):
table = pa.Table.from_pandas(df)
bos = pa.BufferOutputStream()
pq.write_table(table, bos)
- ref_df = pq.read_pandas(
- bos.getvalue(), use_legacy_dataset=use_legacy_dataset).to_pandas()
+ ref_df = pq.read_pandas(bos.getvalue()).to_pandas()
assert isinstance(ref_df.index, pd.CategoricalIndex)
assert ref_df.index.equals(df.index)
@pytest.mark.pandas
-@parametrize_legacy_dataset
-def test_categorical_order_survives_roundtrip(use_legacy_dataset):
+@pytest.mark.dataset
Review Comment:
I suppose that many of those tests also don't need the dataset mark
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -938,46 +671,26 @@ def test_read_multiple_files(tempdir, use_legacy_dataset):
(dirpath / '_SUCCESS.crc').touch()
def read_multiple_files(paths, columns=None, use_threads=True, **kwargs):
- dataset = pq.ParquetDataset(
- paths, use_legacy_dataset=use_legacy_dataset, **kwargs)
+ dataset = pq.ParquetDataset(paths, **kwargs)
return dataset.read(columns=columns, use_threads=use_threads)
result = read_multiple_files(paths)
expected = pa.concat_tables(test_data)
assert result.equals(expected)
- # Read with provided metadata
- # TODO(dataset) specifying metadata not yet supported
Review Comment:
We actually do support that through a different dataset API (`pyarrow.dataset.parquet_dataset`), but that is not wired up in the parquet API. Given that we never got any complaints about this, I suppose fine to just remove (and it should already be tested in the dataset tests, so should be fine to remove the tests here as well)
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -1123,18 +819,14 @@ def _make_example_multifile_dataset(base_path, nfiles=10, file_nrows=5):
return paths
-def _assert_dataset_paths(dataset, paths, use_legacy_dataset):
- if use_legacy_dataset:
- assert set(map(str, paths)) == {x.path for x in dataset._pieces}
- else:
- paths = [str(path.as_posix()) for path in paths]
- assert set(paths) == set(dataset._dataset.files)
+def _assert_dataset_paths(dataset, paths):
+ paths = [str(path.as_posix()) for path in paths]
+ assert set(paths) == set(dataset._dataset.files)
Review Comment:
```suggestion
assert set(paths) == set(dataset.files)
```
(this is exposed nowadays, so we can use the public attribute here in the tests as well)
##########
python/pyarrow/tests/parquet/test_pandas.py:
##########
@@ -150,52 +144,22 @@ def test_pandas_parquet_2_0_roundtrip_read_pandas_no_index_written(
tm.assert_frame_equal(df, df_read)
-# TODO(dataset) duplicate column selection actually gives duplicate columns now
Review Comment:
Do you know if this is actually tested elsewhere? (that you get duplicate columns)
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -752,17 +682,15 @@ def test_fastparquet_cross_compatibility(tempdir):
tm.assert_frame_equal(table_fp.to_pandas(), df)
-@parametrize_legacy_dataset
@pytest.mark.parametrize('array_factory', [
lambda: pa.array([0, None] * 10),
lambda: pa.array([0, None] * 10).dictionary_encode(),
lambda: pa.array(["", None] * 10),
lambda: pa.array(["", None] * 10).dictionary_encode(),
])
-@pytest.mark.parametrize('use_dictionary', [False, True])
Review Comment:
Ah, because it was just not used in practice
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -1230,29 +915,22 @@ def test_ignore_custom_prefixes(tempdir, use_legacy_dataset):
partition_cols=['_part'])
read = pq.read_table(
- tempdir, use_legacy_dataset=use_legacy_dataset,
- ignore_prefixes=['_private'])
+ tempdir, ignore_prefixes=['_private'])
assert read.equals(table)
-@parametrize_legacy_dataset_fixed
-def test_empty_directory(tempdir, use_legacy_dataset):
- # ARROW-5310 - reading empty directory
- # fails with legacy implementation
+def test_empty_directory(tempdir):
Review Comment:
```suggestion
def test_empty_directory(tempdir):
# ARROW-5310 - reading empty directory
```
Maybe keep the issue reference (that's still relevant for the dataset case, as that was the reason this test was introduced)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #39112:
URL: https://github.com/apache/arrow/pull/39112#discussion_r1434133176
##########
python/pyarrow/tests/parquet/test_dataset.py:
##########
@@ -986,31 +699,24 @@ def read_multiple_files(paths, columns=None, use_threads=True, **kwargs):
t = pa.Table.from_pandas(bad_apple)
_write_table(t, bad_apple_path)
- if not use_legacy_dataset:
- # TODO(dataset) Dataset API skips bad files
- return
+ # TODO(dataset) Dataset API skips bad files
Review Comment:
Opened: https://github.com/apache/arrow/issues/39338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1882722210
The `wide-dataframe` case seems a genuine perf regression (and not a flaky outlier as the other listed cases). That might mean that for wide dataframes, the new code path is slower compared to the legacy dataset reader (since with this commit, also when specifying `use_legacy_dataset=True`, the new code path will be used).
That seems to match with the timing in the `use_legacy_dataset=False` case of the wide-dataframe benchmark, as now both benchmarks more or less show the same timing.
However, I can't reproduce this locally with pyarrow 14.0 (where the legacy reader still exists):
```
import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
dataframe = pd.DataFrame(np.random.rand(100, 10000))
table = pa.Table.from_pandas(dataframe)
pq.write_table(table, "test_wide_dataframe.parquet")
```
```
In [7]: %timeit -r 50 pq.read_table("test_wide_dataframe.parquet", use_legacy_dataset=True)
392 ms ± 4.67 ms per loop (mean ± std. dev. of 50 runs, 1 loop each)
In [8]: %timeit -r 50 pq.read_table("test_wide_dataframe.parquet", use_legacy_dataset=False)
350 ms ± 11.5 ms per loop (mean ± std. dev. of 50 runs, 1 loop each)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866626993
@github-actions crossbow submit *hdfs*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866477757
Revision: 77b4ecbb574704d8b8f6ac2acc46fea7eb5d4e02
Submitted crossbow builds: [ursacomputing/crossbow @ actions-bb763d7a57](https://github.com/ursacomputing/crossbow/branches/all?query=actions-bb763d7a57)
|Task|Status|
|----|------|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/7290018285/job/19865878124)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/7290018281/job/19865878141)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866472807
@github-actions crossbow submit *hdfs*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1866926410
Thanks a lot @AlenkaF for the work here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1867435951
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit b70ad0b8801d9ca0634c1937df1fc02c1609548e.
There were 7 benchmark results indicating a performance regression:
- Commit Run on `ursa-i9-9960x` at [2023-12-22 00:21:42Z](https://conbench.ursa.dev/compare/runs/a0418277f7384e6e856e32aa3b630cdd...f6758b0025e84f73a5dbb9ace9b10582/)
- [`file-read` (R) with compression=uncompressed, dataset=fanniemae_2016Q4, file_type=feather, language=R, output_type=dataframe](https://conbench.ursa.dev/compare/benchmarks/06584ba1d1517c668000bd870f2b7237...06584e47ef76787780008bb4e367c322)
- [`wide-dataframe` (Python) with use_legacy_dataset=true](https://conbench.ursa.dev/compare/benchmarks/06584b81a271726b80008631ff39edd0...06584e2469b9746c8000012773fb826f)
- and 5 more (see the report linked below)
The [full Conbench report](https://github.com/apache/arrow/runs/19888738686) has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1862412515
I think the minimal builds are not in the "python" group for crossbow (maybe something we should fix), so I think we have to trigger them separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1862413668
@github-actions crossbow submit example-python-minimal-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-31303: [Python] Remove the legacy ParquetDataset custom python-based implementation [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #39112:
URL: https://github.com/apache/arrow/pull/39112#issuecomment-1854194605
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org