You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by "earlev4 (via GitHub)" <gi...@apache.org> on 2023/02/02 18:49:58 UTC

[GitHub] [arrow] earlev4 opened a new issue, #34010: PyArrow Dataset Schema Infer Depth - Maximum Number of Rows

earlev4 opened a new issue, #34010:
URL: https://github.com/apache/arrow/issues/34010

### Describe the usage question you have. Please include as many useful details as possible.

Hello! First off, a huge thank you to the contributors of the Arrow project! I am grateful for the project.

Please forgive me if this question has been asked before and addressed. I tried to do my due diligence in researching the issue (read PyArrow documentation - Tabular Datasets, Python API, Python Cookbook, etc. and also searched GitHub issues) before asking this question. Perhaps, I missed it. I am using the Python API and specifically the `dataset()` function to read a collection of CSV files. From reading the documentation, I understand by not providing a schema to the `schema` parameter in the `dataset()` function, the schema is inferred. Unfortunately, it appears that the schema inference is not going deep enough into the CSV file to detect the appropriate data types due to missing values in the first thousands of rows. I understand that I can define a schema and pass that to the `schema` parameter, however with 50, 100, or more columns, defining a schema can be extensive.

Here is an example:
```python
my_ds = ds.dataset('./path/to/csv/files', format='csv')
my_ds.head(5)
```

Error example:
```python
ArrowInvalid: In CSV column #100: Row #5000: CSV conversion error to null: invalid value 'ABC'.
```

I believe the error occurs because the rows used to infer the schema determine that column 100 has null values and when row 5000 is reached with a string value of `ABC`, the value is determined invalid.

I know that libraries such as `polars` provide a `infer_schema_length` parameter for the [`read_csv`](https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.read_csv.html) function, where the maximum number of rows to read to infer schema can be defined.

Is there something similar in PyArrow, where the maximum number of rows can be defined to infer the schema for the `dataset` function?

Thanks in advance! I sincerely appreciate any guidance.

### Component(s)

Python

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] westonpace closed issue #34010: [Python] Dataset Schema Infer Depth - Maximum Number of Rows

Posted by "westonpace (via GitHub)" <gi...@apache.org>.

westonpace closed issue #34010: [Python] Dataset Schema Infer Depth - Maximum Number of Rows 
URL: https://github.com/apache/arrow/issues/34010


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] westonpace commented on issue #34010: [Python] Dataset Schema Infer Depth - Maximum Number of Rows

Posted by "westonpace (via GitHub)" <gi...@apache.org>.

westonpace commented on issue #34010:
URL: https://github.com/apache/arrow/issues/34010#issuecomment-1414553376

   It is not obvious but it is possible.  The `block_size` (in [`pyarrow.csv.ReadOptions`](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) is what actually determines our inference depth).  In order to specify a custom read options you will need to create a [`pyarrow.dataset.CsvFileFormat`](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.CsvFileFormat.html#pyarrow.dataset.CsvFileFormat).
   
   Regrettably the inference depth is always somewhat tied in with our I/O performance.  However, I suspect you can bump up the default quite a bit before you start to notice significant effects.
   
    A complete example:
   
   ```
   import pyarrow as pa
   import pyarrow.csv as csv
   import pyarrow.dataset as ds
   
   MiB = 1024*1024
   
   read_options = csv.ReadOptions(block_size=16*MiB) # Note, the default is 1MiB                                                                                                                                      
   csv_format = ds.CsvFileFormat(read_options=read_options)
   
   my_dataset = ds.dataset('/tmp/my_dataset', format=csv_format)
   print(my_dataset.to_table())
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] earlev4 commented on issue #34010: [Python] Dataset Schema Infer Depth - Maximum Number of Rows

Posted by "earlev4 (via GitHub)" <gi...@apache.org>.

earlev4 commented on issue #34010:
URL: https://github.com/apache/arrow/issues/34010#issuecomment-1416593565

   Hi @westonpace. Thank you very, very much! I sincerely appreciate the detailed response. It was very helpful and with your excellent example, I was able to overcome the issue. Thanks again! I am grateful for the help and your time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org