You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2022/10/20 08:22:00 UTC
[jira] [Comment Edited] (ARROW-17308) ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API
[ https://issues.apache.org/jira/browse/ARROW-17308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620323#comment-17620323 ]
Joris Van den Bossche edited comment on ARROW-17308 at 10/20/22 8:21 AM:
-------------------------------------------------------------------------
Yes, the new Dataset API doesn't yet support the {{validate_schema}} keyword, but the work is in progress and is tracked with ARROW-8221 (as it is a duplicated Jira will be closing this issue).
The default in {{ParquetDataset}} is still the legacy API and so the keyword {{validate_schema}} should work (without supplying the new [FileSystem|https://arrow.apache.org/docs/dev/python/generated/pyarrow.fs.FileSystem.html#pyarrow.fs.FileSystem], see: [https://github.com/apache/arrow/blob/962121062e4b13c148f24a6d4fa4b1a2f1be0d88/python/pyarrow/parquet/core.py#L1717-L1718], which is relevant for 9.x and I think it was the same in 6.x, could be wrong though)
was (Author: alenkaf):
Yes, the new Dataset API doesn't yet support the {{validate_schema}} keyword, but the work is in progress and is tracked with https://issues.apache.org/jira/browse/ARROW-8221 (as it is a duplicated Jira will be closing this issue).
The default in {{ParquetDataset}} is still the legacy API and so the keyword {{validate_schema}} should work (without supplying the new [FileSystem|https://arrow.apache.org/docs/dev/python/generated/pyarrow.fs.FileSystem.html#pyarrow.fs.FileSystem], see: [https://github.com/apache/arrow/blob/962121062e4b13c148f24a6d4fa4b1a2f1be0d88/python/pyarrow/parquet/core.py#L1717-L1718], which is relevant for 9.x and I think it was the same in 6.x, could be wrong though)
> ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API
> -----------------------------------------------------------------------------------
>
> Key: ARROW-17308
> URL: https://issues.apache.org/jira/browse/ARROW-17308
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Abderrahmane Jaidi
> Priority: Major
> Labels: dataset-parquet-legacy, dataset-parquet-read
>
> Documentation for PyArrow 6.x and 7.x both indicate that the `validate_schema` argument is supported in the `ParquetDataset` class. Yet passing that argument to an instance results in:
> ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API
> Code:
> {code:python}
> parquet_dataset = pyarrow.parquet.ParquetDataset(
> path_or_paths=paths,
> validate_schema=validate_schema,
> filesystem=filesystem,
> partitioning=partitioning,
> use_legacy_dataset=False,
> ){code}
> Docs link:
> [https://arrow.apache.org/docs/6.0/python/generated/pyarrow.parquet.ParquetDataset.html]
> [https://arrow.apache.org/docs/7.0/python/generated/pyarrow.parquet.ParquetDataset.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)