You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/30 22:57:34 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #14286: ARROW-15838: [R] Coalesce join keys in full outer join
wjones127 opened a new pull request, #14286:
URL: https://github.com/apache/arrow/pull/14286
Also:
* Added support for `keep` argument in join.
* Cleaned up the C++ code for the join so we directly accept the enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r988373692
##########
r/tests/testthat/test-dplyr-join.R:
##########
@@ -141,21 +141,25 @@ test_that("Error handling", {
# TODO: casting: int and float columns?
test_that("right_join", {
- compare_dplyr_binding(
- .input %>%
- right_join(to_join, by = "some_grouping") %>%
- collect(),
- left
- )
+ for (keep in c(TRUE, FALSE)) {
+ compare_dplyr_binding(
+ .input %>%
+ right_join(to_join, by = "some_grouping", keep = keep) %>%
+ collect(),
+ left
+ )
+ }
})
test_that("inner_join", {
- compare_dplyr_binding(
- .input %>%
- inner_join(to_join, by = "some_grouping") %>%
- collect(),
- left
- )
+ for (keep in c(TRUE, FALSE)) {
+ compare_dplyr_binding(
+ .input %>%
+ inner_join(to_join, by = "some_grouping") %>%
+ collect(),
+ left
+ )
+ }
Review Comment:
Yes, that was an issue. I fixed that just before I requested your review. It should be good now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson merged pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
nealrichardson merged PR #14286:
URL: https://github.com/apache/arrow/pull/14286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14286:
URL: https://github.com/apache/arrow/pull/14286#issuecomment-1280232825
['Python', 'R'] benchmarks have high level of regressions.
[test-mac-arm](https://conbench.ursa.dev/compare/runs/6f574fc52662402abe540b9f3fef615c...84ed67fddc40434ba07cf2ec4ece36c1/)
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/3c9f6b44f74f49d7b8cbe210ca1018cd...9bd397c6fe704230b2c2c0364e26d324/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14286:
URL: https://github.com/apache/arrow/pull/14286#issuecomment-1264114876
https://issues.apache.org/jira/browse/ARROW-15838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thisisnic commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
thisisnic commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r988811346
##########
r/tests/testthat/test-dplyr-join.R:
##########
@@ -338,3 +342,33 @@ test_that("arrow dplyr query can join two datasets", {
}
)
})
+
+test_that("joins on datasets handles keep", {
+ full_data_df <- tibble::tibble(
+ x = rep(c("a", "b"), each = 5),
+ y = rep(1:5, 2),
+ z = rep("zzz", 10),
+ index = 1:10
+ )
+ small_dataset_df <- tibble::tibble(
+ value = c(0.1, 0.2, 0.3, 0.4, 0.5),
+ x = c(rep("a", 3), rep("b", 2)),
+ y = 1:5,
+ z = 6:10
+ )
+ full_data <- InMemoryDataset$create(full_data_df)
Review Comment:
Just went to check the original repro, and can confirm that the behaviour is the same for Tables and Datasets, so using a Table is fine here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r988355355
##########
r/tests/testthat/test-dplyr-join.R:
##########
@@ -338,3 +342,33 @@ test_that("arrow dplyr query can join two datasets", {
}
)
})
+
+test_that("joins on datasets handles keep", {
+ full_data_df <- tibble::tibble(
+ x = rep(c("a", "b"), each = 5),
+ y = rep(1:5, 2),
+ z = rep("zzz", 10),
+ index = 1:10
+ )
+ small_dataset_df <- tibble::tibble(
+ value = c(0.1, 0.2, 0.3, 0.4, 0.5),
+ x = c(rep("a", 3), rep("b", 2)),
+ y = 1:5,
+ z = 6:10
+ )
+ full_data <- InMemoryDataset$create(full_data_df)
Review Comment:
You are correct. Nic's originaly repro involved writing out a Parquet dataset, so I thought the dataset part was important to it. I was able to still repro by simplifying to an InMemoryDataset, but didn't think to try Table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r991354505
##########
r/R/query-engine.R:
##########
@@ -384,3 +397,65 @@ needs_projection <- function(projection, schema) {
!identical(field_names, names(projection)) || # Any fields are renamed
!identical(field_names, names(schema)) # The fields are reordered
}
+
+#' Create projection needed to coalesce join keys after a full outer join
+#'
+#' @examples
+#' test_join <- list(
+#' type = JoinType$FULL_OUTER,
+#' right_data = arrow_table(x = 1, y = 2, z = "x"),
+#' by = c("x", "y"),
+#' suffix = c(".x", ".y"),
+#' keep = FALSE
+#' )
+#' post_join_projection(c("value", "x", "y", "z"), test_join)
+#'
+#' @noRd
+post_join_projection <- function(left_names, join_config) {
+ # Need to coalesce the key columns
+ right_names <- names(join_config$right_data)
+ # Collect mapping of which columns on left need to be coalesced with which
+ # column on the right side.
+ coalesce_targets <- data.frame(
Review Comment:
Rather than a data.frame, would a named list or character vector be a simpler map to work with?
##########
r/R/dplyr-collect.R:
##########
@@ -123,15 +123,25 @@ implicit_schema <- function(.data) {
# Add cols from right side, except for semi/anti joins
right_cols <- .data$join$right_data$selected_columns
left_cols <- .data$selected_columns
- right_fields <- map(
- right_cols[setdiff(names(right_cols), .data$join$by)],
- ~ .$type(.data$join$right_data$.data$schema)
- )
- # get right table and left table column names excluding the join key
- right_cols_ex_by <- right_cols[setdiff(names(right_cols), .data$join$by)]
- left_cols_ex_by <- left_cols[setdiff(names(left_cols), .data$join$by)]
- # find the common column names in left and right tables
- common_cols <- intersect(names(right_cols_ex_by), names(left_cols_ex_by))
+
+ # If keep = TRUE, we want to keep the key columns in the RHS. Otherwise,
+ # they will be dropped.
+ if (.data$join$keep) {
+ # find the common column names in left and right tables
+ common_cols <- intersect(names(right_cols), names(left_cols))
+ right_fields <- map(right_cols, ~ .$type(.data$join$right_data$.data$schema))
+ } else {
+ right_fields <- map(
+ right_cols[setdiff(names(right_cols), .data$join$by)],
+ ~ .$type(.data$join$right_data$.data$schema)
+ )
+ # get right table and left table column names excluding the join key
Review Comment:
```suggestion
# get right table and left table column projections excluding the join key(s)
```
##########
r/src/compute-exec.cpp:
##########
@@ -404,37 +403,16 @@ std::shared_ptr<compute::ExecNode> ExecNode_Join(
for (auto&& name : left_output) {
left_out_refs.emplace_back(std::move(name));
}
- if (type != 0 && type != 2) {
+ // dplyr::semi_join => LEFT_SEMI; dplyr::anti_join => LEFT_ANTI
+ // So ignoring RIGHT_SEMI and RIGHT_ANTI here.
Review Comment:
```suggestion
// So ignoring RIGHT_SEMI and RIGHT_ANTI here because dplyr doesn't implement them.
```
##########
r/R/query-engine.R:
##########
@@ -165,6 +172,12 @@ ExecPlan <- R6Class("ExecPlan",
left_suffix = .data$join$suffix[[1]],
right_suffix = .data$join$suffix[[2]]
)
+ # If we are doing a full outer join and not keeping the join keys of
Review Comment:
Would it make sense to move this projection step into `full_join.arrow_dplyr_query()`? I could see arguments either way, but it might be cleaner to move the logic there so that there is less special handling in `ExecPlan$build()`, so that it can more simply just render the query plan it is given.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r985000060
##########
r/src/arrow_types.h:
##########
@@ -38,6 +38,8 @@
#include <arrow/dataset/type_fwd.h>
#endif
+#include <arrow/compute/exec/options.h>
Review Comment:
This was moved over here so that `compute::JoinType` can be passed from R instead of having to pass an integer (and then converting within C++).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r994969297
##########
r/R/query-engine.R:
##########
@@ -165,6 +172,12 @@ ExecPlan <- R6Class("ExecPlan",
left_suffix = .data$join$suffix[[1]],
right_suffix = .data$join$suffix[[2]]
)
+ # If we are doing a full outer join and not keeping the join keys of
Review Comment:
We could do that, but I would have to yank `left_output` and `right_output` resolution up to `do_join`, since those are coupled with `keep` and the full outer coalesce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r995138972
##########
r/R/query-engine.R:
##########
@@ -384,3 +397,65 @@ needs_projection <- function(projection, schema) {
!identical(field_names, names(projection)) || # Any fields are renamed
!identical(field_names, names(schema)) # The fields are reordered
}
+
+#' Create projection needed to coalesce join keys after a full outer join
+#'
+#' @examples
+#' test_join <- list(
+#' type = JoinType$FULL_OUTER,
+#' right_data = arrow_table(x = 1, y = 2, z = "x"),
+#' by = c("x", "y"),
+#' suffix = c(".x", ".y"),
+#' keep = FALSE
+#' )
+#' post_join_projection(c("value", "x", "y", "z"), test_join)
+#'
+#' @noRd
+post_join_projection <- function(left_names, join_config) {
+ # Need to coalesce the key columns
+ right_names <- names(join_config$right_data)
+ # Collect mapping of which columns on left need to be coalesced with which
+ # column on the right side.
+ coalesce_targets <- data.frame(
Review Comment:
I'm not sure how that would work. If there's something like a Python dictionary that can represent `Dict[int, int]` that would be nice. But it would only eliminate one line I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r995140513
##########
r/R/query-engine.R:
##########
@@ -165,6 +172,12 @@ ExecPlan <- R6Class("ExecPlan",
left_suffix = .data$join$suffix[[1]],
right_suffix = .data$join$suffix[[2]]
)
+ # If we are doing a full outer join and not keeping the join keys of
Review Comment:
I just pushed a refactor commit that does that. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14286:
URL: https://github.com/apache/arrow/pull/14286#issuecomment-1280232679
Benchmark runs are scheduled for baseline = 81e1fbc1dec3639f228a50282fc0a4d657c18884 and contender = ee1f76308433e377435b82d9b1ddf8e164316f16. ee1f76308433e377435b82d9b1ddf8e164316f16 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/ebcccf7fd6d94dda822628ebd4556029...f388567e84a541daadadb3f4894cf651/)
[Failed :arrow_down:0.0% :arrow_up:15.56%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/6f574fc52662402abe540b9f3fef615c...84ed67fddc40434ba07cf2ec4ece36c1/)
[Finished :arrow_down:0.54% :arrow_up:6.52%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/3c9f6b44f74f49d7b8cbe210ca1018cd...9bd397c6fe704230b2c2c0364e26d324/)
[Finished :arrow_down:0.32% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/1f70d4dacbc0457ea5911a8090c9132d...1e35c5cbc74c465bbb5dcf215dfc835d/)
Buildkite builds:
[Finished] [`ee1f7630` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1714)
[Failed] [`ee1f7630` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1733)
[Finished] [`ee1f7630` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1716)
[Finished] [`ee1f7630` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1727)
[Finished] [`81e1fbc1` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1713)
[Failed] [`81e1fbc1` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1732)
[Finished] [`81e1fbc1` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1715)
[Finished] [`81e1fbc1` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1726)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #14286: ARROW-15838: [R] Coalesce join keys in full outer join
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #14286:
URL: https://github.com/apache/arrow/pull/14286#discussion_r995765800
##########
r/tests/testthat/test-dplyr-join.R:
##########
@@ -338,3 +346,30 @@ test_that("arrow dplyr query can join two datasets", {
}
)
})
+
+test_that("full joins handle keep", {
+ full_data_df <- tibble::tibble(
+ x = rep(c("a", "b"), each = 5),
+ y = rep(1:5, 2),
+ z = rep("zzz", 10),
+ index = 1:10
+ )
+ small_dataset_df <- tibble::tibble(
+ value = c(0.1, 0.2, 0.3, 0.4, 0.5),
+ x = c(rep("a", 3), rep("b", 2)),
+ y = 1:5,
+ z = 6:10
+ )
+ full_data <- Table$create(full_data_df)
+ small_dataset <- Table$create(small_dataset_df)
+
+ for (keep in c(TRUE, FALSE)) {
+ result <- full_join(small_dataset, full_data, by = c("y", "x"), keep = !!keep) %>%
+ arrange(index) %>%
+ collect()
+ expected <- full_join(small_dataset_df, full_data_df, by = c("y", "x"), keep = !!keep) %>%
+ arrange(index) %>%
+ collect()
+ expect_equal(result, expected)
+ }
Review Comment:
You should be able to use compare_dplyr_binding here too. The data you're joining will get converted to Table automatically
```suggestion
for (keep in c(TRUE, FALSE)) {
compare_dplyr_binding(
.input %>%
full_join(full_data_df, by = c("y", "x"), keep = !!keep) %>%
arrange(index) %>%
collect(),
small_dataset_df
)
}
```
##########
r/R/dplyr-join.R:
##########
@@ -83,7 +94,17 @@ full_join.arrow_dplyr_query <- function(x,
suffix = c(".x", ".y"),
...,
keep = FALSE) {
- do_join(x, y, by, copy, suffix, ..., keep = keep, join_type = "FULL_OUTER")
+ query <- do_join(x, y, by, copy, suffix, ..., keep = keep, join_type = "FULL_OUTER")
+
+ # If we are doing a full outer join and not keeping the join keys of
+ # both sides, we need to coalesce. Otherwise, rows that exist in the
+ # RHS will have NAs for the join keys.
+ if (!keep) {
+ query$selected_columns <- post_join_projection(names(x), names(y), handle_join_by(by, x, y), suffix)
+ collapse.arrow_dplyr_query(query)
Review Comment:
I don't think you need this collapse() here since do_join() collapses for you and you're just projecting after. (Moreover, since you aren't assigning the result here and return `query` below, the result of this collapse isn't being used.)
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org