You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Marc Garcia (Jira)" <ji...@apache.org> on 2020/10/29 15:09:00 UTC

[jira] [Created] (ARROW-10419) Add max_rows parameter to pyarrow.csv.ReadOptions

Marc Garcia created ARROW-10419:
-----------------------------------

             Summary: Add max_rows parameter to pyarrow.csv.ReadOptions
                 Key: ARROW-10419
                 URL: https://issues.apache.org/jira/browse/ARROW-10419
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Python
            Reporter: Marc Garcia


I'm trying to read only the first 1,000 rows of a huge CSV with PyArrow.

I don't see a way to do this with Arrow. I guess it should be easy to implement by adding a `max_rows` parameter to pyarrow.csv.ReadOptions.

After reading the first 1,000, it should be possible to load the next 1,000 (or any other chunk) by using both the new `max_rows` together with `skip_rows` (e.g. `pyarrow.csv.read_csv(path, pyarrow.csv.ReadOption(skip_rows=1_000, max_rows=1_000)` would read from 1,000 to 2,000).

Thanks!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)