You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "marberts (via GitHub)" <gi...@apache.org> on 2026/02/20 05:15:50 UTC
[PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
marberts opened a new pull request, #49343:
URL: https://github.com/apache/arrow/pull/49343
### Rationale for this change
`write_dataset(df)` need not preserve the row-ordering of `df` across partitions. The arrow C++ library was recently updated (since 21.0.0) so that row ordering can be preserved when writing across partitions. This is useful for cases where it is assumed that row-ordering is unchanged within each partition.
``` r
df <- tibble::tibble(x = 1:1.5e6, g = rep(1:15, each = 1e5))
df |>
dplyr::group_by(g) |>
arrow::write_dataset("test1", preserve_order = FALSE)
df |>
dplyr::group_by(g) |>
arrow::write_dataset("test2", preserve_order = TRUE)
test1 <- arrow::open_dataset("test1") |>
dplyr::collect()
test2 <- arrow::open_dataset("test2") |>
dplyr::collect()
# Current behavior.
all.equal(test1 |> sort_by(~ g), df)
#> [1] "Component \"x\": Mean relative difference: 0.0475804"
# Preserve order.
all.equal(test2 |> sort_by(~ g), df)
#> [1] TRUE
```
<sup>Created on 2026-02-20 with [reprex v2.1.1](https://reprex.tidyverse.org)</sup>
### What changes are included in this PR?
Added an argument `preserve_order` to `write_dataset()` that sets `FileSystemDatasetWriteOptions.preserve_order` to true in the call to `ExecPlan_Write()`.
### Are these changes tested?
Partially. The change is small, so I haven't written unit tests. I can revisit this if necessary.
### Are there any user-facing changes?
Yes, there is a new argument in `write_dataset()`. The default keeps the current behavior and the argument appears after all existing arguments, so the change in backwards compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49343:
URL: https://github.com/apache/arrow/pull/49343#issuecomment-3931726479
:warning: GitHub issue #49340 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #49343:
URL: https://github.com/apache/arrow/pull/49343#discussion_r2842745421
##########
r/tests/testthat/test-dataset-write.R:
##########
@@ -1015,3 +1015,20 @@ test_that("Dataset write wrappers can write flat files using readr::write_csv()
c("true", "false", "NOVALUE", "true", "false", "true", "false", "NOVALUE", "true", "false")
)
})
+
+test_that("Row order is preserved when writing large parquet dataset", {
+ skip_if_not_available("parquet")
+ # Make a data frame with a sufficiently large number of rows.
+ df <- data.frame(x = 1:1.1e6)
+ dst_dir1 <- make_temp_dir()
+ write_dataset(df, dst_dir1)
+
+ dst_dir2 <- make_temp_dir()
+ write_dataset(df, dst_dir2, preserve_order = TRUE)
+
+ ds1 <- open_dataset(dst_dir1) |> collect()
+ ds2 <- open_dataset(dst_dir2) |> collect()
+
+ expect_true(any(ds1$x != df$x))
+ expect_true(all(ds2$x == df$x))
Review Comment:
```suggestion
unordered_dir <- make_temp_dir()
write_dataset(df, unordered_dir)
ordered_dir <- make_temp_dir()
write_dataset(df, ordered_dir, preserve_order = TRUE)
unordered_ds <- open_dataset(unordered_dir) |> collect()
ordered_ds2 <- open_dataset(ordered_dir) |> collect()
expect_true(any(unordered_ds$x != df$x))
expect_true(all(ordered_ds$x == df$x))
```
Making the names slightly more understandable / readable.
also what about:
```
# unordered is setequal, but not ordered
expect_setequal(unordered_ds$x, df$x)
expect_false(all(unordered_ds$x == df$x))
# but ordered is exactly equal
expect_equal(ordered_ds$x, df$x)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #49343:
URL: https://github.com/apache/arrow/pull/49343#discussion_r2847236707
##########
r/tests/testthat/test-dataset-write.R:
##########
@@ -1015,3 +1015,26 @@ test_that("Dataset write wrappers can write flat files using readr::write_csv()
c("true", "false", "NOVALUE", "true", "false", "true", "false", "NOVALUE", "true", "false")
)
})
+
+test_that("Row order is preserved when writing large parquet dataset", {
+ skip_if_not_available("parquet")
+ # Make a data frame with a sufficiently large number of rows.
+ df <- data.frame(x = 1:1.1e6)
+
+ unordered_dir <- make_temp_dir()
+ write_dataset(df, unordered_dir)
+
+ ordered_dir <- make_temp_dir()
+ write_dataset(df, ordered_dir, preserve_order = TRUE)
+
+ unordered_ds <- open_dataset(unordered_dir) |> collect()
+ ordered_ds <- open_dataset(ordered_dir) |> collect()
+
+ # Unordered is set equal, but not necessarily equal.
+ expect_setequal(unordered_ds$x, df$x)
+ # expect_false(all(unordered_ds$x == df$x)) can fail on certain
+ # platforms, so is not tested.
Review Comment:
Oh, interesting — which platforms? I can also imagine that there are certain circumstances where it will be happen to wind up in the right order by chance too. It would be good to know where this failed though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane merged PR #49343:
URL: https://github.com/apache/arrow/pull/49343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49343:
URL: https://github.com/apache/arrow/pull/49343#discussion_r2844180340
##########
r/tests/testthat/test-dataset-write.R:
##########
@@ -1015,3 +1015,20 @@ test_that("Dataset write wrappers can write flat files using readr::write_csv()
c("true", "false", "NOVALUE", "true", "false", "true", "false", "NOVALUE", "true", "false")
)
})
+
+test_that("Row order is preserved when writing large parquet dataset", {
+ skip_if_not_available("parquet")
+ # Make a data frame with a sufficiently large number of rows.
+ df <- data.frame(x = 1:1.1e6)
+ dst_dir1 <- make_temp_dir()
+ write_dataset(df, dst_dir1)
+
+ dst_dir2 <- make_temp_dir()
+ write_dataset(df, dst_dir2, preserve_order = TRUE)
+
+ ds1 <- open_dataset(dst_dir1) |> collect()
+ ds2 <- open_dataset(dst_dir2) |> collect()
+
+ expect_true(any(ds1$x != df$x))
+ expect_true(all(ds2$x == df$x))
Review Comment:
You're right, your suggestion for naming and test assertions is clearer. Thanks!
It turns out the unordered case is platform dependent, as the test is failing in one case on windows: https://github.com/apache/arrow/actions/runs/22290480076/job/64536892096?pr=49343#step:10:24418.
I updated the test to only test for equality in the ordered case, noting that the test for lack of equality for the unordered case can fail. I don't think this is a problem because it's testing the desired behavior when `preserve_order = TRUE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49343:
URL: https://github.com/apache/arrow/pull/49343#discussion_r2844180340
##########
r/tests/testthat/test-dataset-write.R:
##########
@@ -1015,3 +1015,20 @@ test_that("Dataset write wrappers can write flat files using readr::write_csv()
c("true", "false", "NOVALUE", "true", "false", "true", "false", "NOVALUE", "true", "false")
)
})
+
+test_that("Row order is preserved when writing large parquet dataset", {
+ skip_if_not_available("parquet")
+ # Make a data frame with a sufficiently large number of rows.
+ df <- data.frame(x = 1:1.1e6)
+ dst_dir1 <- make_temp_dir()
+ write_dataset(df, dst_dir1)
+
+ dst_dir2 <- make_temp_dir()
+ write_dataset(df, dst_dir2, preserve_order = TRUE)
+
+ ds1 <- open_dataset(dst_dir1) |> collect()
+ ds2 <- open_dataset(dst_dir2) |> collect()
+
+ expect_true(any(ds1$x != df$x))
+ expect_true(all(ds2$x == df$x))
Review Comment:
You're right, your suggestion for naming and test assertions is clearer. Thanks!
It turns out the unordered case is platform dependent, as the test is failing in one case on windows: https://github.com/apache/arrow/actions/runs/22290480076/job/64536892096?pr=49343#step:10:24418.
I updated the test to only test for equality in the ordered case, noting that the test for lack of equality for the unordered case can fail. I don't think this is a problem because it's still testing the desired behavior when `preserve_order = TRUE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on code in PR #49343:
URL: https://github.com/apache/arrow/pull/49343#discussion_r2844180340
##########
r/tests/testthat/test-dataset-write.R:
##########
@@ -1015,3 +1015,20 @@ test_that("Dataset write wrappers can write flat files using readr::write_csv()
c("true", "false", "NOVALUE", "true", "false", "true", "false", "NOVALUE", "true", "false")
)
})
+
+test_that("Row order is preserved when writing large parquet dataset", {
+ skip_if_not_available("parquet")
+ # Make a data frame with a sufficiently large number of rows.
+ df <- data.frame(x = 1:1.1e6)
+ dst_dir1 <- make_temp_dir()
+ write_dataset(df, dst_dir1)
+
+ dst_dir2 <- make_temp_dir()
+ write_dataset(df, dst_dir2, preserve_order = TRUE)
+
+ ds1 <- open_dataset(dst_dir1) |> collect()
+ ds2 <- open_dataset(dst_dir2) |> collect()
+
+ expect_true(any(ds1$x != df$x))
+ expect_true(all(ds2$x == df$x))
Review Comment:
You're right, your suggestion for naming and test assertions is clearer. Thanks!
It turns out the unordered case is platform dependent, as the test is failing in one case on windows: https://github.com/apache/arrow/actions/runs/22290480076/job/64536892096?pr=49343#step:10:24418.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "marberts (via GitHub)" <gi...@apache.org>.
marberts commented on PR #49343:
URL: https://github.com/apache/arrow/pull/49343#issuecomment-3953666388
Awesome, thanks!
For your question, it seems like `write_dataset()` behaves differently on windows and ubuntu. In some cases row-order is kept on windows when `preserve_order = FALSE` (but not always). We can see this from the CI job when I had the test `expect_false(all(unordered_ds$x == df$x))`: it passed on ubuntu and windows with an older version of R, but failed on windows with a newer version of R (https://github.com/apache/arrow/actions/runs/22290480076/job/64535896168). I could reproduce this on my machine with R 4.5.2, with both ubuntu and windows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49340: [R] Preserve row order in `write_dataset()` [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #49343:
URL: https://github.com/apache/arrow/pull/49343#issuecomment-3954869472
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 376afb888368c252767aa249af06e7ae23c60f7a.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/64749916031) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org