[jira] [Updated] (ARROW-8058) [C++][Python][Dataset] Provide an option to toggle validation and schema inference in FileSystemDatasetFactoryOptions

["Francois Saint-Jacques (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Francois Saint-Jacques updated ARROW-8058:
------------------------------------------
    Summary: [C++][Python][Dataset] Provide an option to toggle validation and schema inference in FileSystemDatasetFactoryOptions  (was: [C++][Python][Dataset] Provide an option to skip validation in FileSystemDatasetFactoryOptions)

> [C++][Python][Dataset] Provide an option to toggle validation and schema inference in FileSystemDatasetFactoryOptions
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-8058
>                 URL: https://issues.apache.org/jira/browse/ARROW-8058
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++ - Dataset, Python
>    Affects Versions: 0.16.0
>            Reporter: Ben Kietzman
>            Priority: Major
>             Fix For: 1.0.0
>
>
> This can be costly and is not always necessary.
> At the same time we could move file validation into the scan tasks; currently all files are inspected as the dataset is constructed, which can be expensive if the filesystem is slow. We'll be performing the validation multiple times but the check will be cheap since at scan time we'll be reading the file into memory anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)