You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/02/19 18:42:27 UTC
[GitHub] [arrow] dhicks opened a new issue #12469: [R] int32/int64 issues in opening CSVs
dhicks opened a new issue #12469:
URL: https://github.com/apache/arrow/issues/12469
Background: I'm trying to use arrow to (a) assemble two large sets of CSVs into two (folders of) parquet files and then (b) combine the two into a single working dataset. The two sets of CSVs all have the same column structure: `article_id` (string), `phrase` (string), `n` (integer). Without specifying a schema, in step (a) `open_dataset` parses `n` as `int64` in the first set and as `int32` in the second set. This results in an error when trying to combine them (because, I guess, automatic casting from int32 to int64 isn't supported yet). The maximum value of `n` across the entire first set is 576, so I'm not sure why arrow thinks it needs 64 bits in the first place.
Shorter background: In a collection of CSVs, `open_dataset` parses the column `n` as `int64` and I need it to be `int32`.
An example CSV, the first file (from the first of the two sets), is attached: [1960-1-01.csv.zip](https://github.com/apache/arrow/files/8102736/1960-1-01.csv.zip)
I've tried specifying a schema, as follows:
```
text_ar = open_dataset(noun_phrase_dir,
format = 'csv',
schema = schema(article_id = string(),
phrase = string(),
n = int32()))
```
However, when specifying a schema, `open_dataset` includes the header row, and raises an error because the letter n can't be cast to an integer:
```
Error: Invalid: Could not open CSV input source [path to file]: Invalid: In CSV column #2: Row #1: CSV conversion error to int32: invalid value 'n'
```
So then I tried adding the argument `skip = 1`:
```
text_ar = open_dataset(noun_phrase_dir,
format = 'csv',
schema = schema(article_id = string(),
phrase = string(),
n = int32()),
skip = 1)
```
Which returns this error:
```
Error: The following option is supported in "read_delim_arrow" functions but not yet supported here: "skip"
```
My next approach was to let `open_dataset` just use the default scheme, and cast `n` to `int32` as a second step. I can't seem to figure out how to fit `call_function` into `dplyr` syntax, with a few variations on the following:
```
text_ar |>
dplyr::mutate(n = call_function("cast", n, int32()))
```
```
Error: Expression call_function("cast", n, int32()) not supported in Arrow
Call collect() first to pull data into R.
```
The `Table` class has a `$cast` method that might do the trick. But `text_ar` is a `FileSystemDataset` and either doesn't have a `$cast` method or I don't understand what it expects.
```
text_ar$cast(schema(n = int32()))
```
```
Error: attempt to apply non-function
```
It does have a `$schema` format, but per the docs this doesn't support casting:
```
text_ar$schema <- schema(n = int32())
```
```
Error: Type error: fields had matching names but differing types. From: n: int64 To: n: int32
```
I haven't been able to find any information on converting a `FileSystemDataset` to a `Table`. I have no idea where to go from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dhicks commented on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
dhicks commented on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046285026
Thank you! FWIW `compute()` seems to consistently crash; I'm not sure whether this is because it's not really designed for treating local files as though they were a remote database, but `collect()` works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] dhicks closed issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
dhicks closed issue #12469:
URL: https://github.com/apache/arrow/issues/12469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eitsupi edited a comment on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
eitsupi edited a comment on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046246626
The following code may be helpful.
```R
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- open_dataset("1960-1-01.csv", format = "csv")
ds |>
mutate(n = cast(n, int32())) |>
write_dataset("tmp_dir", format = "parquet")
open_dataset("tmp_dir")
```
I don't think I would have noticed the existence of `cast()` if I hadn't touched pyarrow, since it doesn't exist as a stand-alone function and is currently barely mentioned in the help pages, which can be used in dplyr calculations.
It's definitely worth improving the documentation.
> I haven't been able to find any information on converting a `FileSystemDataset` to a `Table`. I have no idea where to go from here.
`dplyr::compute()` can be used in the `dplyr` pipeline.
This is in line with the general usage of `dbplyr`, but it certainly seems to be barely mentioned in the arrow documentation.
By the way, I recently did the same thing with pyarrow, but first I read each CSV file with its exact type and converted it to a Parquet file, then I read a collection of Parquet files and wrote them out again in a partitioned dataset.
The Parquet to Parquet conversion was very easy, as the read/write speed is much faster once converted to Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eitsupi edited a comment on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
eitsupi edited a comment on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046246626
The following code may be helpful.
```R
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- open_dataset("1960-1-01.csv", format = "csv")
ds |>
mutate(n = cast(n, int32())) |>
write_dataset("tmp_dir", format = "parquet")
open_dataset("tmp_dir")
```
I don't think I would have noticed the existence of `cast()` if I hadn't touched pyarrow, since it doesn't exist as a stand-alone function and is currently barely mentioned in the help pages, which can be used in `dplyr` calculations.
It's definitely worth improving the documentation.
> I haven't been able to find any information on converting a `FileSystemDataset` to a `Table`. I have no idea where to go from here.
`dplyr::compute()` can be used in the `dplyr` pipeline.
This is in line with the general usage of `dbplyr`, but it certainly seems to be barely mentioned in the arrow documentation.
By the way, I recently did the same thing with pyarrow, but first I read each CSV file with its exact type and converted it to a Parquet file, then I read a collection of Parquet files and wrote them out again in a partitioned dataset.
The Parquet to Parquet conversion was very easy, as the read/write speed is much faster once converted to Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046287019
@dhicks if you happen to have a reprex for the crash, it would be much appreciated - compute() shouldn't crash on local files. (Is it for the same file you've attached?)
Also, what version of Arrow are you using? (`arrow_info()` would give you this.)
You can use `col_types` or `skip_rows`. The documentation is a little unclear on this point, but the acceptable option names actually come from CsvReadOptions/CsvParseOptions/CsvConvertOptions. For example:
```r
> open_dataset('./temp/1960-1-01.csv', format='csv', schema=schema(article_id=string(), phrase=string(), n=int32())) %>% collect()
Error: Invalid: Could not open CSV input source '/home/lidavidm/temp/1960-1-01.csv': Invalid: In CSV column #2: Row #1: CSV conversion error to int32: invalid value 'n'
> open_dataset('./temp/1960-1-01.csv', format='csv', schema=schema(article_id=string(), phrase=string(), n=int32()), skip_rows=1) %>% collect()
# A tibble: 452 × 3
article_id phrase n
<chr> <chr> <int>
1 1960-1-01 it 63
2 1960-1-01 we 24
3 1960-1-01 world 13
4 1960-1-01 numbers 11
5 1960-1-01 they 11
6 1960-1-01 our_numbers 10
7 1960-1-01 he 9
8 1960-1-01 life 9
9 1960-1-01 i 8
10 1960-1-01 mankind 7
# … with 442 more rows
> open_dataset('./temp/1960-1-01.csv', format='csv', col_types=schema(article_id=string(), phrase=string(), n=int32())) %>% collect()
# A tibble: 452 × 3
article_id phrase n
<chr> <chr> <int>
1 1960-1-01 it 63
2 1960-1-01 we 24
3 1960-1-01 world 13
4 1960-1-01 numbers 11
5 1960-1-01 they 11
6 1960-1-01 our_numbers 10
7 1960-1-01 he 9
8 1960-1-01 life 9
9 1960-1-01 i 8
10 1960-1-01 mankind 7
# … with 442 more rows
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046861027
Adding to what has been said there by @lidavidm, `skip_rows` is the way to go here. However, `skip` should also work, and I have opened a ticket on the project JIRA (where we track bugs/features) to make this work properly: https://issues.apache.org/jira/browse/ARROW-15743
Hopefully things should work using `skip_rows`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eitsupi commented on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
eitsupi commented on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046246626
The following code may be helpful.
```R
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
ds <- open_dataset("1960-1-01.csv", format = "csv")
ds |>
mutate(n = cast(n, int32())) |>
write_dataset("tmp_dir", format = "parquet")
open_dataset("tmp_dir")
```
> I haven't been able to find any information on converting a `FileSystemDataset` to a `Table`. I have no idea where to go from here.
`dplyr::compute()` can be used in the `dplyr` pipeline.
This is in line with the general usage of `dbplyr`, but it certainly seems to be barely mentioned in the arrow documentation.
By the way, I recently did the same thing with pyarrow, but first I read each CSV file with its exact type and converted it to a Parquet file, then I read a collection of Parquet files and wrote them out again in a partitioned dataset.
The Parquet to Parquet conversion was very easy, as the read/write speed is much faster once converted to Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic edited a comment on issue #12469: [R] int32/int64 issues in opening CSVs
Posted by GitBox <gi...@apache.org>.
thisisnic edited a comment on issue #12469:
URL: https://github.com/apache/arrow/issues/12469#issuecomment-1046861027
Adding to what has been said there by @lidavidm, `skip_rows` is the way to go here. However, `skip` should also work, and I have opened a ticket on the project JIRA (where we track bugs/features) to make this work properly: https://issues.apache.org/jira/browse/ARROW-15743
Hopefully things should work using `skip_rows` - let us know if not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org