You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Jonathan Keane (Jira)" <ji...@apache.org> on 2022/03/04 15:31:00 UTC
[jira] [Resolved] (ARROW-15743) [R] `skip` not connected up to `skip_rows` on open_dataset despite error messages indicating otherwise
[ https://issues.apache.org/jira/browse/ARROW-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Keane resolved ARROW-15743.
------------------------------------
Fix Version/s: 8.0.0
Resolution: Fixed
Issue resolved by pull request 12523
[https://github.com/apache/arrow/pull/12523]
> [R] `skip` not connected up to `skip_rows` on open_dataset despite error messages indicating otherwise
> ------------------------------------------------------------------------------------------------------
>
> Key: ARROW-15743
> URL: https://issues.apache.org/jira/browse/ARROW-15743
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Reporter: Nicola Crane
> Assignee: Nicola Crane
> Priority: Major
> Labels: pull-request-available
> Fix For: 8.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> If I open a dataset of CSVs with a schema, the error message tells me to supply {{`skip = 1`}} if my data contains a header row (to prevent it being read in as data), but only {{skip_rows = 1}} actually works.
> {code:r}
> library(arrow)
> library(dplyr)
> td <- tempfile()
> dir.create(td)
> write_dataset(mtcars, td, format = "csv")
> schema <- schema(mpg = float64(), cyl = float64(), disp = float64(), hp = float64(),
> drat = float64(), wt = float64(), qsec = float64(), vs = float64(),
> am = float64(), gear = float64(), carb = float64())
> open_dataset(td, format = "csv", schema = schema) %>%
> collect()
> #> Error in `handle_csv_read_error()`:
> #> ! Invalid: Could not open CSV input source '/tmp/RtmppZbpeF/file6cec135ed29c/part-0.csv': Invalid: In CSV column #0: Row #1: CSV conversion error to double: invalid value 'mpg'
> #> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:550 decoder_.Decode(data, size, quoted, &value)
> #> /home/nic2/arrow/cpp/src/arrow/csv/parser.h:123 status
> #> /home/nic2/arrow/cpp/src/arrow/csv/converter.cc:554 parser.VisitColumn(col_index, visit)
> #> /home/nic2/arrow/cpp/src/arrow/csv/reader.cc:463 arrow::internal::UnwrapOrRaise(maybe_decoded_arrays)
> #> /home/nic2/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:445 iterator_.Next()
> #> /home/nic2/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
> #> /home/nic2/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)
> #> ℹ If you have supplied a schema and your data contains a header row, you should supply the argument `skip = 1` to prevent the header being read in as data.
> open_dataset(td, format = "csv", schema = schema, skip = 1) %>%
> collect()
> #> Error: The following option is supported in "read_delim_arrow" functions but not yet supported here: "skip"
> open_dataset(td, format = "csv", schema = schema, skip_rows = 1) %>%
> collect()
> #> # A tibble: 32 × 11
> #> mpg cyl disp hp drat wt qsec vs am gear carb
> #> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
> #> 1 21 6 160 110 3.9 2.62 16.5 0 1 4 4
> #> 2 21 6 160 110 3.9 2.88 17.0 0 1 4 4
> #> 3 22.8 4 108 93 3.85 2.32 18.6 1 1 4 1
> #> 4 21.4 6 258 110 3.08 3.22 19.4 1 0 3 1
> #> 5 18.7 8 360 175 3.15 3.44 17.0 0 0 3 2
> #> 6 18.1 6 225 105 2.76 3.46 20.2 1 0 3 1
> #> 7 14.3 8 360 245 3.21 3.57 15.8 0 0 3 4
> #> 8 24.4 4 147. 62 3.69 3.19 20 1 0 4 2
> #> 9 22.8 4 141. 95 3.92 3.15 22.9 1 0 4 2
> #> 10 19.2 6 168. 123 3.92 3.44 18.3 1 0 4 4
> #> # … with 22 more rows
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)