You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/05 15:36:42 UTC
[GitHub] [arrow] toppyy opened a new pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
toppyy opened a new pull request #12083:
URL: https://github.com/apache/arrow/pull/12083
Supplement CsvReadOptions with column_names from schema if column_names are not set in read_options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r791814292
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,21 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs schema given via read_options should raise an error
+ opts <- CsvReadOptions$create(column_names = c("i", "d"))
+ schema <- schema(int = int32(), dbl = float64())
+
+ expect_error(
+ open_dataset(csv_dir, format = "csv", schema = schema, read_options = opts),
+ "column_names not matching or not found in schema-names"
+ )
Review comment:
Thanks again for all of the comments, really helpful!
At the moment the example you provided would *not* raise an error if I've got it right. This is due to the following logic in `csv_file_format_read_opts`:
https://github.com/apache/arrow/blob/0fa4b9ca1e7b13ac230f85075d86d783eeea6a74/r/R/dataset-format.R#L223-L225
If schema is set, it overrides `column_names` in all situations. Because the check is done after `csv_file_format_read_opts` no error is raised despite of conflicting arguments from the user (which might be a little confusing for the user?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1014835853
> you're welcome to submit extra examples with the docs as part of this PR or we can open up a separate issue for that?
I think I'd prefer a separate issue just for the sake of clarity on my end.
Thanks for the issue for the cookbook. I'll get back to that later after this one is sorted :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1012386125
I think that makes a lot sense! Trying to infer the column names from schema added (a surprising) amount of complexity to the code for little to none added value for the user. As you say, they can pass the arguments using a different approach.
Just to make sure I got it right before I jump into making changes, the approach we're taking is this:
Instead of deriving the column_names from the schema, we raise an error if `CsvReadOptions$create()` is used for read_options and it has no column names or they differ from the schema?
If so, could the check for column names be inside `CsvFileFormat$create`? As the issue is limited to formats that use this function (not relevant for parquet).
I can write this up in the documentation as a part of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1023422739
Benchmark runs are scheduled for baseline = 816c986753dc1b7fd85934d703ad4d88abe38424 and contender = 81c6dad564b0c1afa87741a7c9a94dedc0ad9411. 81c6dad564b0c1afa87741a7c9a94dedc0ad9411 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/0c27ef4935394179b66bd27adabb3785...0864b560066440c5b68998f28362b596/)
[Failed :arrow_down:8.21% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7310ef8212cc4467982a3143a896430d...7d259632d61d4d07a7334272a3133689/)
[Finished :arrow_down:0.43% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9d34a3a71fa34995b0de813509a70456...3093f2bfc887486f96e5caf9c1643363/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1007915879
Thanks for copy-pasting the discussion - it didn't occur to me. I tried option 2, but ended up having to change the class CsvReadOptions also (see below why). So the latest commits really implement options one and two.
To me it still seems that it's impossible to get the value of _skip_rows_ later on if user has set it like this:
> `open_dataset(
> td,
> format = 'csv',
> read_options = arrow::CsvReadOptions$create(
> skip_rows = 1
> )
> )`
Because of this I had to implement CsvReadOptions$skip_rows to be able to retrieve the value in `csv_file_format_read_opts`. I could be wrong though.
The functionality is as follows:
- Check if column_names are given as a separate argument. If so, use them.
- Check if column_names are given in read_options. If so, use them.
- Check if schema is set. If so, use names from schema as column names.
For skip rows, we do the same (with the exception of using schema).
The code is a bit verbose and may be improved by some simplification. Also, if any other options (in addition to column_names & skip_rows) are introduced, it needs an update - so not ideal in that sense.
Let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1007915879
Thanks for copy-pasting the discussion - it didn't occur to me. I tried option 2, but ended up having to change the class CsvReadOptions also (see below why). So the latest commits really implement options one and two.
To me it still seems that it's impossible to get the value of _skip_rows_ later on if user has set it like this:
> `open_dataset(
> td,
> format = 'csv',
> read_options = arrow::CsvReadOptions$create(
> skip_rows = 1
> )
> )`
Because of this I had to implement CsvReadOptions$skip_rows to be able to retrieve the value in `csv_file_format_read_opts`. I could be wrong though.
The functionality is as follows:
- Check if column_names are given as a separate argument. If so, use them.
- Check if column_names are given in read_options. If so, use them.
- Check if schema is set. If so, use names from schema as column names.
For skip rows, we do the same (with the exception of using schema).
The code is a bit verbose and may be improved by some simplification. Also, if any other options (in addition to column_names & skip_rows) are introduced, it needs an update - so not ideal in that sense.
Let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1013924539
> It seems to that the `open_dataset` docs (or the cookbook) might benefit from additional examples or a separate section (like in read_csv_arrow) on this issue?
Agreed - you're welcome to submit extra examples with the docs as part of this PR or we can open up a separate issue for that? Let me know which you'd rather do.
The cookbook is in a separate [repo](https://github.com/apache/arrow-cookbook), so I've opened a ticket there: https://github.com/apache/arrow-cookbook/issues/120
At present there is no chapter on datasets - I was thinking of starting one soon, but if you'd like to get it started, or contribute some content once it's started, let me know on that ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r790663925
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,21 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs schema given via read_options should raise an error
+ opts <- CsvReadOptions$create(column_names = c("i", "d"))
+ schema <- schema(int = int32(), dbl = float64())
Review comment:
This is slightly unusual spacing here, and I notice that the linter is complaining about a "Missing terminal newline" a bit further down; would you mind running the styler on the files you have changed?
Instructions here: https://arrow.apache.org/docs/dev/r/articles/developers/workflow.html
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,21 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs schema given via read_options should raise an error
+ opts <- CsvReadOptions$create(column_names = c("i", "d"))
+ schema <- schema(int = int32(), dbl = float64())
+
+ expect_error(
+ open_dataset(csv_dir, format = "csv", schema = schema, read_options = opts),
+ "column_names not matching or not found in schema-names"
+ )
Review comment:
Thanks for the test, looking good. One minor suggestion: how about swapping this for a simpler case, e.g.:
```
open_dataset(csv_dir, format = "csv", schema = schema, column_names = c("these", "are", "not", "the", "columns", "from", "the", "schema"))
```
It then removes the need to do the more complex work of `opts <- CsvReadOptions$create(column_names = c("i", "d"))` which is technically off-label usage that we don't recommend anyway.
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,31 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
+
+ abort(c(
+ paste(
+ "column_names not matching or not found in schema-names:",
+ deparse1(c(mismatch_colnames, not_in_schema))
+ ),
+ i = "Set column_names to match names of schema",
+ i = "Omit the column_names argument",
+ i = "Omit the read_options argument"
Review comment:
I really love that you've come up with helpful solutions for the end-user here. However, I have a concern that given that there are multiple paths that users can take to get here, and `read_options` isn't an argument that the user is necessarily aware of (it's a few levels down in a function they wouldn't be likely to call directly), and so this error message might be a little confusing - it may be sufficient just to state what's wrong, without any suggestions.
Could I get your thoughts here @jonkeane?
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,31 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
+
+ abort(c(
+ paste(
+ "column_names not matching or not found in schema-names:",
Review comment:
What do you think to the idea of rephrasing this to use "should" or "must" as per the tidyverse style guide (https://style.tidyverse.org/error-messages.html)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1006618779
Just pasting here the conversation from JIRA:
> I did, however, run into trouble. Say, for example, the user has set skip_rows-option like this:
`read_options=arrow::CsvReadOptions$create(skip_rows=1))`
I image we'd like to keep whatever options the user has set when we re-create the `CsvReadOptions` object with column names from the schema. The problem is that I cannot access `skip_rows` in the object after it's created, so I cannot use that information to create another instance of `CsvReadOptions` that has both the `column_names` and `skip_rows` set (plus any other options).
> Any thoughts? Is there a way to access `skip_rows` and other attributes that I'm unaware of? Of course, one solution is to change class declaration of `CsvReadOptions` to have access to these attributes.
Thanks for opening this draft PR! After checking out a copy of your branch, I understand what's going on here and why it's not working a lot better not.
Currently, the approaches that come to mind for me are:
1. as you say, update that class so we can modify contents after creation
2. update the signature of `CsvFileFormat$create()`, setting `read_options` to a default value of `NULL` and then calling `csv_file_format_read_opts()` later in the function, passing in both the schema and any user-set values.
I haven't fully fleshed out the second option or tested to see if it'll work, but if it does I'd be in favour of doing it that way so we can make the change we need without having to modify the structure of the existing classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r791842872
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,31 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
+
+ abort(c(
+ paste(
+ "column_names not matching or not found in schema-names:",
+ deparse1(c(mismatch_colnames, not_in_schema))
+ ),
+ i = "Set column_names to match names of schema",
+ i = "Omit the column_names argument",
+ i = "Omit the read_options argument"
Review comment:
I agree with @thisisnic, I really like solutions provided in error messages generally. But here, since there are so many ways one can get to this (and not all of the solutions would work in all of those paths), it's probably better to simply error with the problem up above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r791842872
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,31 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
+
+ abort(c(
+ paste(
+ "column_names not matching or not found in schema-names:",
+ deparse1(c(mismatch_colnames, not_in_schema))
+ ),
+ i = "Set column_names to match names of schema",
+ i = "Omit the column_names argument",
+ i = "Omit the read_options argument"
Review comment:
I agree with @thisisnic, I really like solutions provided in error messages generally. But here, since there are so many ways one can get to this (and not all of the solutions would work in all of those paths), it's probably better to simply error with the problem up above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1012889685
Yeah, spot on, and there is an extra level of complexity added here due to the fact that the way in which we infer column names from a schema is a bit incorrect (but this is being dealt with as part of another issue)
Are you happy with leaving the existing code which derives column names from a schema as it was before any of the changes in this PR, and as you say, raise an error if CsvReadOptions$create() is used for read_options but is not consistent with the schema?
There's a PR which was just merged that does a similar thing but relates to partitioning, and is a great example of this kind of thing being done really nicely: https://github.com/apache/arrow/blob/99f7c3cf3e6c2a9555ceff3d48ef73e485ede546/r/R/dataset-factory.R#L85-L95
What you said about where the code for this should go sounds right and thanks for volunteering to update the docs too!
Thanks for sticking with this even though we've drastically changed what we're doing to resolve this - even though we're not using the code from your previous solutions, the process of testing them out and the surrounding discussion has been really helpful for identifying some serious enhancements that can be made to how the schema and column name components interact and also how we direct our users to work with `open_dataset()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1011224682
Thanks for the updates here @toppyy . I've taken the time to have a proper think about this, and on reflection, I don't think we need to make `open_dataset( td, format = 'csv', read_options = CsvReadOptions$create( skip_rows = 1 ))` work for users as they can pass in their `skip_rows` parameter this way: `open_dataset( td, format = 'csv', skip_rows = 1)`.
Directly using `CsvReadOptions$create()` is pretty low-level and so here we can probably assume that someone using it is responsible for making sure things match up themselves (though should absolutely add further documentation to show our recommended way that users work with `open_dataset()` so it's clear - perhaps that could be part of this PR if you're interested but no worries if not).
I feel an alternative solution here might be just to check if there are conflicting arguments specified to `open_dataset()` (e.g. through specifying the `read_options` argument in the ellipses as well as individual read options). It might be something along the lines of adding validation at the end of open_dataset so that if `(!is.null(schema))` and format is csv, ensure that `identical(names(schema), read_options$column_names)` or raise an error.
Does that make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r785474474
##########
File path: r/R/dataset.R
##########
@@ -123,6 +123,7 @@
#' or call [`$NewScan()`][Scanner] to construct a query directly.
#' @export
#' @seealso `vignette("dataset", package = "arrow")`
+#' See [read_csv_arrow()] on how to specify column names and types for "csv"/"text" and "tsv" -formats.
Review comment:
This is great, though I think it might fit better after the content on line 121 along with links to `read_feather()` and `read_parquet()` and a more generic comment about viewing those page for format-specific options.
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,18 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ if (length(read_options$column_names) > 0 & !is.null(schema) & !identical(names(schema), read_options$column_names)) {
+ abort(c(
+ '"column_names" in read_options do not match the schema.',
+ i = "Set column_names in read_options to match the schema",
+ i = "Omit the read_options argument"
+ ))
Review comment:
Looking good, just a couple of suggestions:
- the user of the `open_dataset()` function won't necessarily have used the function `read_options()` directly, so we should remove the reference to it to avoid confusion
- to provide maximal useful information to the end-user, could you print out the arguments which are mismatches between `column_names` and the schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1023422739
Benchmark runs are scheduled for baseline = 816c986753dc1b7fd85934d703ad4d88abe38424 and contender = 81c6dad564b0c1afa87741a7c9a94dedc0ad9411. 81c6dad564b0c1afa87741a7c9a94dedc0ad9411 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/0c27ef4935394179b66bd27adabb3785...0864b560066440c5b68998f28362b596/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7310ef8212cc4467982a3143a896430d...7d259632d61d4d07a7334272a3133689/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9d34a3a71fa34995b0de813509a70456...3093f2bfc887486f96e5caf9c1643363/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r786252402
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,18 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ if (length(read_options$column_names) > 0 & !is.null(schema) & !identical(names(schema), read_options$column_names)) {
+ abort(c(
+ '"column_names" in read_options do not match the schema.',
+ i = "Set column_names in read_options to match the schema",
+ i = "Omit the read_options argument"
+ ))
Review comment:
Thanks for the good comments!
It raises an error and prints out the mismatches between read_options$column_names and names(schema).
At the moment a mismatch is either due to 1) set difference or a 2) different order of column names. However, I'm a bit unsure of the latter. Do we in fact want to raise an error if the order of names differ? For example: `c("a","b") != c("a","b")`
Also, if this is the case, I'd be interested if you have any tips on convenient ways to do this. I resorted to using `suppressWarnings` to handle cases where vector lengths differ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1005827714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r791814292
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,21 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs schema given via read_options should raise an error
+ opts <- CsvReadOptions$create(column_names = c("i", "d"))
+ schema <- schema(int = int32(), dbl = float64())
+
+ expect_error(
+ open_dataset(csv_dir, format = "csv", schema = schema, read_options = opts),
+ "column_names not matching or not found in schema-names"
+ )
Review comment:
Thanks again for all of the comments, really helpful!
At the moment the example you provided would *not* raise an error if I've got it right. This is due to the following logic in `csv_file_format_read_opts`:
https://github.com/apache/arrow/blob/0fa4b9ca1e7b13ac230f85075d86d783eeea6a74/r/R/dataset-format.R#L223-L225
If schema is set, it overrides `column_names` in all situations. Because the check is done after `csv_file_format_read_opts` no error is raised despite of conflicting arguments from the user (which might be a little confusing for the user?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r793747392
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,27 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ # Evaluate opts first to catch any unsupported arguments
+ force(opts)
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+
Review comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r785542810
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,18 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ if (length(read_options$column_names) > 0 & !is.null(schema) & !identical(names(schema), read_options$column_names)) {
+ abort(c(
+ '"column_names" in read_options do not match the schema.',
+ i = "Set column_names in read_options to match the schema",
+ i = "Omit the read_options argument"
+ ))
Review comment:
Also, please could you add a test which triggers this error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r786252402
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,18 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ if (length(read_options$column_names) > 0 & !is.null(schema) & !identical(names(schema), read_options$column_names)) {
+ abort(c(
+ '"column_names" in read_options do not match the schema.',
+ i = "Set column_names in read_options to match the schema",
+ i = "Omit the read_options argument"
+ ))
Review comment:
Thanks for the good comments!
It raises an error and prints out the mismatches between read_options$column_names and names(schema).
At the moment a mismatch is either due to 1) set difference or a 2) different order of column names. However, I'm a bit unsure of the latter. Do we in fact want to raise an error if the order of names differ? For example: `c("a","b") != c("b","a")`
Also, if this is the case, I'd be interested if you have any tips on convenient ways to do this. I resorted to using `suppressWarnings` to handle cases where vector lengths differ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r791822069
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,21 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs schema given via read_options should raise an error
+ opts <- CsvReadOptions$create(column_names = c("i", "d"))
+ schema <- schema(int = int32(), dbl = float64())
+
+ expect_error(
+ open_dataset(csv_dir, format = "csv", schema = schema, read_options = opts),
+ "column_names not matching or not found in schema-names"
+ )
Review comment:
Good catch - let's update that to something like `if (!is.null(schema)) && is.null(opts[["column_names"]])`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1023422739
Benchmark runs are scheduled for baseline = 816c986753dc1b7fd85934d703ad4d88abe38424 and contender = 81c6dad564b0c1afa87741a7c9a94dedc0ad9411. 81c6dad564b0c1afa87741a7c9a94dedc0ad9411 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/0c27ef4935394179b66bd27adabb3785...0864b560066440c5b68998f28362b596/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7310ef8212cc4467982a3143a896430d...7d259632d61d4d07a7334272a3133689/)
[Finished :arrow_down:0.43% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9d34a3a71fa34995b0de813509a70456...3093f2bfc887486f96e5caf9c1643363/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1023422739
Benchmark runs are scheduled for baseline = 816c986753dc1b7fd85934d703ad4d88abe38424 and contender = 81c6dad564b0c1afa87741a7c9a94dedc0ad9411. 81c6dad564b0c1afa87741a7c9a94dedc0ad9411 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/0c27ef4935394179b66bd27adabb3785...0864b560066440c5b68998f28362b596/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7310ef8212cc4467982a3143a896430d...7d259632d61d4d07a7334272a3133689/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9d34a3a71fa34995b0de813509a70456...3093f2bfc887486f96e5caf9c1643363/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic closed pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic closed pull request #12083:
URL: https://github.com/apache/arrow/pull/12083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1013892092
I reverted the previous commits and implemented the following logic in `CsvFileFormat$create` as discussed: If a schema is specified and column names are set in read_options, we raise an error if the two do not match.
Also added a note of this in the documentation.
I thought about the documentation aspect a bit. As a starting point, I added link to [read_csv_arrow](https://arrow.apache.org/docs/r/reference/read_delim_arrow.html) into `open_dataset` docs. It has some explanation on how to specify column names and types. Though I suspect this might be a bit misleading for the readers who are unaware of the fact that both functions end up calling `CsvReadOptions$create`.
It seems to that the `open_dataset` docs (or the cookbook) might benefit from additional examples or a separate section (like in read_csv_arrow) on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic edited a comment on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic edited a comment on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1011224682
Thanks for the updates here @toppyy . I've taken the time to have a proper think about this, and on reflection, I don't think we need to make `open_dataset( td, format = 'csv', read_options = CsvReadOptions$create( skip_rows = 1 ))` work for users as they can pass in their `skip_rows` parameter this way: `open_dataset( td, format = 'csv', skip_rows = 1)`.
Directly using `CsvReadOptions$create()` is pretty low-level and so here we can probably assume that someone using it is responsible for making sure things match up themselves (though should absolutely add further documentation to show our recommended way that users work with `open_dataset()` so it's clear - perhaps that could be part of this PR if you're interested but no worries if not).
I feel an alternative solution here might be just to check if there are conflicting arguments specified to `open_dataset()` (e.g. through specifying the `read_options` argument in the ellipses as well as individual read options). It might be something along the lines of adding validation at the end of open_dataset so that if `(!is.null(schema))` and format is csv, ensure that `identical(names(schema), read_options$column_names)` or raise an error.
How does that sound?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] toppyy commented on pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
toppyy commented on pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#issuecomment-1019446733
I refactored/simplified the column_names vs. schema-names comparison a bit.
While doing this, I realized that the solution did not solve the original issue. I was comparing column names versus schema only when both are set. However, the original issue presented itself when read_options was set _without_ column_names. Like so:
`open_dataset(
td,
format='csv',
schema = diamond_schema,
skip_rows = 1,
read_options=arrow::CsvReadOptions$create(
skip_rows = 1
# ..and no column_names
)
) %>% collect()`
To raise an error in situation also, the code now checks that `identical(read_options$column_names,names(schema))` in all scenarios where schema is set. So the example would create an error despite the fact that `column_names` is not set at all. I added a hint for this situation in the error message ("Omit the read_options -argument").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r793688513
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,30 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ # Evaluate opts first to catch any unsupported arguments
+ force(opts)
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
Review comment:
```suggestion
```
Removing these as no longer needed due to changes below
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,20 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs. schema-names should raise an error
+ schema <- schema(int = int32(), dbl = float64())
+
+ expect_error(
+ open_dataset(csv_dir, format = "csv", schema = schema, column_names = c("these", "wont", "match")),
+ "column_names must match schema-names"
Review comment:
```suggestion
"Values in `column_names` must match schema field names"
```
Updating this to match change made above
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,30 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ # Evaluate opts first to catch any unsupported arguments
+ force(opts)
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
+
+ abort(
+ paste(
+ "column_names must match schema-names:",
+ deparse1(c(mismatch_colnames, not_in_schema))
+ )
Review comment:
```suggestion
paste(
"Values in `column_names` must match schema field names"
)
```
##########
File path: r/R/dataset-format.R
##########
@@ -122,6 +122,30 @@ CsvFileFormat$create <- function(...,
opts = csv_file_format_parse_options(...),
convert_options = csv_file_format_convert_opts(...),
read_options = csv_file_format_read_opts(...)) {
+
+ # Evaluate opts first to catch any unsupported arguments
+ force(opts)
+
+ options <- list(...)
+ schema <- options[["schema"]]
+
+ column_names <- read_options$column_names
+ schema_names <- names(schema)
+
+ if (!is.null(schema) & !identical(schema_names, column_names)) {
+
+ # Element wise comparison and set differnce of column_names and names in schema
+ mismatch_colnames <- column_names[match(column_names, schema_names, nomatch = 0) != seq(column_names)]
+ not_in_schema <- setdiff(schema_names, column_names)
+
+ abort(
+ paste(
+ "column_names must match schema-names:",
+ deparse1(c(mismatch_colnames, not_in_schema))
+ )
Review comment:
I've just removed the pasting of the fields as it's a little unclear which are missing from which, so best to leave out to reduce ambiguity!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a change in pull request #12083: ARROW-14744: [R] open_dataset() error when `schema` argument supplied, but `column_names` not supplied to `CSVReadOptions`
Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #12083:
URL: https://github.com/apache/arrow/pull/12083#discussion_r791822069
##########
File path: r/tests/testthat/test-dataset-csv.R
##########
@@ -299,3 +299,21 @@ test_that("open_dataset() deals with BOMs (byte-order-marks) correctly", {
tibble(a = c(1, 3), b = c(2, 4))
)
})
+
+test_that("Error if read_options$column_names and schema-names differ (ARROW-14744)", {
+
+ dst_dir <- make_temp_dir()
+ dst_file <- file.path(dst_dir, "file.csv")
+ df <- df1[, c("int", "dbl")]
+ write.csv(df, dst_file, row.names = FALSE, quote = FALSE)
+
+ # Mismatch of column names vs schema given via read_options should raise an error
+ opts <- CsvReadOptions$create(column_names = c("i", "d"))
+ schema <- schema(int = int32(), dbl = float64())
+
+ expect_error(
+ open_dataset(csv_dir, format = "csv", schema = schema, read_options = opts),
+ "column_names not matching or not found in schema-names"
+ )
Review comment:
Good catch - let's update that to something like `if (!is.null(schema)) && is.null(opts[["column_names"]])`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org