You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/19 06:11:47 UTC
[GitHub] [arrow] vibhatha opened a new pull request #12185: Arrow-15020: [R] Add bindings for new dataset writing options [WIP]
vibhatha opened a new pull request #12185:
URL: https://github.com/apache/arrow/pull/12185
Adding R bindings for `write_dataset` with options `max_open_files`, `max_rows_per_file`, `min_rows_per_group` and `max_rows_per_group`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1024374093
@jonkeane thank you for the detailed review 👍
I will address them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033280519
Benchmark runs are scheduled for baseline = 7018a4baf16845d538aee9259000dc1a3ea3f3ab and contender = 0a56006d6184ce9a65056cd7952d168853ecf0b2. 0a56006d6184ce9a65056cd7952d168853ecf0b2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a051f3e283234eeead23c56d8f0e6df5...011061dc9d9748438e7e591c80c37a5e/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/cf954642e7ca4941b75dc1f34b6ee437...34302865f4d44128a7a1dd16f5bfbc1d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/1d37475cb17e47849694f1623da13533...a08c736b334941cb8417609fbff7ad84/)
[Finished :arrow_down:0.17% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/53bb44b13c75410db9687f379297b05d...1ae573c952d948a2818ef35b06750eb0/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha edited a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha edited a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1026424005
@jonkeane it seems like the style issue is not coming from the files edited in this PR.
But I could be wrong. Could you please take a look: https://github.com/apache/arrow/runs/5015536685?check_suite_focus=true#step:6:2037
Also it could have been introduced when I did the `devtools::document()`, it modified bunch of other files which seem not related to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha removed a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha removed a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1025961158
>
@jonkeane I tried `devtools::document()` and the resultant diff is the following. I am adding the diff, because it edited a set of files. Just want to make sure whether I followed it right.
```git
diff --git a/r/DESCRIPTION b/r/DESCRIPTION
index ae4bbcb8c..c7daff152 100644
--- a/r/DESCRIPTION
+++ b/r/DESCRIPTION
@@ -38,7 +38,7 @@ Imports:
utils,
vctrs
Roxygen: list(markdown = TRUE, r6 = FALSE, load = "source")
-RoxygenNote: 7.1.2
+RoxygenNote: 7.1.2.9000
Config/testthat/edition: 3
VignetteBuilder: knitr
Suggests:
diff --git a/r/NAMESPACE b/r/NAMESPACE
index d841bb290..6233b4fcd 100644
--- a/r/NAMESPACE
+++ b/r/NAMESPACE
@@ -153,6 +153,8 @@ export(JsonTableReader)
export(LargeListArray)
export(ListArray)
export(LocalFileSystem)
+export(MapArray)
+export(MapType)
export(MemoryMappedFile)
export(MessageReader)
export(MessageType)
@@ -251,6 +253,7 @@ export(list_flights)
export(list_of)
export(load_flight_server)
export(map_batches)
+export(map_of)
export(match_arrow)
export(matches)
export(mmap_create)
diff --git a/r/man/FileFormat.Rd b/r/man/FileFormat.Rd
index cabacc937..3c6fd330b 100644
--- a/r/man/FileFormat.Rd
+++ b/r/man/FileFormat.Rd
@@ -40,7 +40,7 @@ you encounter one that \code{arrow} should support. Also, the following options
supported. From \link{CsvReadOptions}:
\itemize{
\item \code{skip_rows}
-\item \code{column_names}
+\item \code{column_names}. Note that if a \link{Schema} is specified, \code{column_names} must match those specified in the schema.
\item \code{autogenerate_column_names}
From \link{CsvFragmentScanOptions} (these values can be overridden at scan time):
\item \code{convert_options}: a \link{CsvConvertOptions}
diff --git a/r/man/array.Rd b/r/man/array.Rd
index 81109a8f7..0b1371a56 100644
--- a/r/man/array.Rd
+++ b/r/man/array.Rd
@@ -9,6 +9,7 @@
\alias{ListArray}
\alias{LargeListArray}
\alias{FixedSizeListArray}
+\alias{MapArray}
\alias{StructScalar}
\title{Arrow Arrays}
\description{
diff --git a/r/man/data-type.Rd b/r/man/data-type.Rd
index 69475d8ad..dbf0436f7 100644
--- a/r/man/data-type.Rd
+++ b/r/man/data-type.Rd
@@ -38,6 +38,8 @@
\alias{large_list_of}
\alias{FixedSizeListType}
\alias{fixed_size_list_of}
+\alias{MapType}
+\alias{map_of}
\title{Apache Arrow data types}
\usage{
int8()
@@ -109,6 +111,8 @@ list_of(type)
large_list_of(type)
fixed_size_list_of(type, list_size)
+
+map_of(key_type, item_type, .keys_sorted = FALSE)
}
\arguments{
\item{byte_width}{byte width for \code{FixedSizeBinary} type.}
diff --git a/r/man/open_dataset.Rd b/r/man/open_dataset.Rd
index 025ce5fd1..eef18b06e 100644
--- a/r/man/open_dataset.Rd
+++ b/r/man/open_dataset.Rd
@@ -81,7 +81,7 @@ it is assumed to be "text".}
\item{...}{additional arguments passed to \code{dataset_factory()} when \code{sources}
is a directory path/URI or vector of file paths/URIs, otherwise ignored.
These may include \code{format} to indicate the file format, or other
-format-specific options.}
+format-specific options (see \code{\link[=read_csv_arrow]{read_csv_arrow()}}, \code{\link[=read_parquet]{read_parquet()}} and \code{\link[=read_feather]{read_feather()}} on how to specify these).}
}
\value{
A \link{Dataset} R6 object. Use \code{dplyr} methods on it to query the data,
diff --git a/r/man/write_dataset.Rd b/r/man/write_dataset.Rd
index 3c01396a1..5bcfef2b9 100644
--- a/r/man/write_dataset.Rd
+++ b/r/man/write_dataset.Rd
@@ -13,6 +13,10 @@ write_dataset(
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
+ max_rows_per_file = 0L,
+ min_rows_per_group = 0L,
+ max_rows_per_group = bitwShiftL(1, 20),
...
)
}
@@ -56,6 +60,25 @@ partitions which data is not written to.
\item{max_partitions}{maximum number of partitions any batch may be
written into. Default is 1024L.}
+\item{max_open_files}{maximum number of files that can be left opened
+during a write operation. If greater than 0 then this will limit the
+maximum number of files that can be left open. If an attempt is made to open
+too many files then the least recently used file will be closed.
+If this setting is set too low you may end up fragmenting your data
+into many small files. The default is 900 which also allows some # of files to be
+open by the scanner before hitting the default Linux limit of 1024.}
+
+\item{max_rows_per_file}{maximum number of rows to be placed in
+any single file}
+
+\item{min_rows_per_group}{write the row groups to the disk when this number of
+rows have accumulated.}
+
+\item{max_rows_per_group}{maximum rows allowed in a single
+group and when this number of rows is exceeded, it is split and the next set
+of rows is written to the next group. This value must be set such that it is
+greater than \code{min_rows_per_group}.}
+
\item{...}{additional format-specific arguments. For available Parquet
options, see \code{\link[=write_parquet]{write_parquet()}}. The available Feather options are:
\itemize{
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on a change in pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#discussion_r794648563
##########
File path: r/R/dataset-write.R
##########
@@ -111,6 +121,10 @@ write_dataset <- function(dataset,
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
Review comment:
It is mentioned specifically in the c++ docs. https://github.com/apache/arrow/blob/c5b757fe607b1e5824053da279a727e35e877e0a/cpp/src/arrow/dataset/file_base.h#L368
It contains a nice description 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha edited a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha edited a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029769235
@kou Sure, I will rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029556800
@jonkeane thank you for looking into this and making an update 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1026424005
@jonkeane it seems like the style issue is not coming from the files edited in this PR.
But I could be wrong. Could you please take a look: https://github.com/apache/arrow/runs/5015536685?check_suite_focus=true#step:6:2037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029766029
Could you rebase on master to resolve the failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12185: Arrow-15020: [R] Add bindings for new dataset writing options [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1016117492
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on a change in pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on a change in pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#discussion_r794656588
##########
File path: r/R/dataset-write.R
##########
@@ -111,6 +121,10 @@ write_dataset <- function(dataset,
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
Review comment:
Of course I will
I missed this in this PR. We did the same for the Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029769235
Sure, I will rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033280519
Benchmark runs are scheduled for baseline = 7018a4baf16845d538aee9259000dc1a3ea3f3ab and contender = 0a56006d6184ce9a65056cd7952d168853ecf0b2. 0a56006d6184ce9a65056cd7952d168853ecf0b2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a051f3e283234eeead23c56d8f0e6df5...011061dc9d9748438e7e591c80c37a5e/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/cf954642e7ca4941b75dc1f34b6ee437...34302865f4d44128a7a1dd16f5bfbc1d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/1d37475cb17e47849694f1623da13533...a08c736b334941cb8417609fbff7ad84/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/53bb44b13c75410db9687f379297b05d...1ae573c952d948a2818ef35b06750eb0/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1025961158
>
@jonkeane I tried `devtools::document()` and the resultant diff is the following. I am adding the diff, because it edited a set of files. Just want to make sure whether I followed it right.
```git
diff --git a/r/DESCRIPTION b/r/DESCRIPTION
index ae4bbcb8c..c7daff152 100644
--- a/r/DESCRIPTION
+++ b/r/DESCRIPTION
@@ -38,7 +38,7 @@ Imports:
utils,
vctrs
Roxygen: list(markdown = TRUE, r6 = FALSE, load = "source")
-RoxygenNote: 7.1.2
+RoxygenNote: 7.1.2.9000
Config/testthat/edition: 3
VignetteBuilder: knitr
Suggests:
diff --git a/r/NAMESPACE b/r/NAMESPACE
index d841bb290..6233b4fcd 100644
--- a/r/NAMESPACE
+++ b/r/NAMESPACE
@@ -153,6 +153,8 @@ export(JsonTableReader)
export(LargeListArray)
export(ListArray)
export(LocalFileSystem)
+export(MapArray)
+export(MapType)
export(MemoryMappedFile)
export(MessageReader)
export(MessageType)
@@ -251,6 +253,7 @@ export(list_flights)
export(list_of)
export(load_flight_server)
export(map_batches)
+export(map_of)
export(match_arrow)
export(matches)
export(mmap_create)
diff --git a/r/man/FileFormat.Rd b/r/man/FileFormat.Rd
index cabacc937..3c6fd330b 100644
--- a/r/man/FileFormat.Rd
+++ b/r/man/FileFormat.Rd
@@ -40,7 +40,7 @@ you encounter one that \code{arrow} should support. Also, the following options
supported. From \link{CsvReadOptions}:
\itemize{
\item \code{skip_rows}
-\item \code{column_names}
+\item \code{column_names}. Note that if a \link{Schema} is specified, \code{column_names} must match those specified in the schema.
\item \code{autogenerate_column_names}
From \link{CsvFragmentScanOptions} (these values can be overridden at scan time):
\item \code{convert_options}: a \link{CsvConvertOptions}
diff --git a/r/man/array.Rd b/r/man/array.Rd
index 81109a8f7..0b1371a56 100644
--- a/r/man/array.Rd
+++ b/r/man/array.Rd
@@ -9,6 +9,7 @@
\alias{ListArray}
\alias{LargeListArray}
\alias{FixedSizeListArray}
+\alias{MapArray}
\alias{StructScalar}
\title{Arrow Arrays}
\description{
diff --git a/r/man/data-type.Rd b/r/man/data-type.Rd
index 69475d8ad..dbf0436f7 100644
--- a/r/man/data-type.Rd
+++ b/r/man/data-type.Rd
@@ -38,6 +38,8 @@
\alias{large_list_of}
\alias{FixedSizeListType}
\alias{fixed_size_list_of}
+\alias{MapType}
+\alias{map_of}
\title{Apache Arrow data types}
\usage{
int8()
@@ -109,6 +111,8 @@ list_of(type)
large_list_of(type)
fixed_size_list_of(type, list_size)
+
+map_of(key_type, item_type, .keys_sorted = FALSE)
}
\arguments{
\item{byte_width}{byte width for \code{FixedSizeBinary} type.}
diff --git a/r/man/open_dataset.Rd b/r/man/open_dataset.Rd
index 025ce5fd1..eef18b06e 100644
--- a/r/man/open_dataset.Rd
+++ b/r/man/open_dataset.Rd
@@ -81,7 +81,7 @@ it is assumed to be "text".}
\item{...}{additional arguments passed to \code{dataset_factory()} when \code{sources}
is a directory path/URI or vector of file paths/URIs, otherwise ignored.
These may include \code{format} to indicate the file format, or other
-format-specific options.}
+format-specific options (see \code{\link[=read_csv_arrow]{read_csv_arrow()}}, \code{\link[=read_parquet]{read_parquet()}} and \code{\link[=read_feather]{read_feather()}} on how to specify these).}
}
\value{
A \link{Dataset} R6 object. Use \code{dplyr} methods on it to query the data,
diff --git a/r/man/write_dataset.Rd b/r/man/write_dataset.Rd
index 3c01396a1..5bcfef2b9 100644
--- a/r/man/write_dataset.Rd
+++ b/r/man/write_dataset.Rd
@@ -13,6 +13,10 @@ write_dataset(
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
+ max_rows_per_file = 0L,
+ min_rows_per_group = 0L,
+ max_rows_per_group = bitwShiftL(1, 20),
...
)
}
@@ -56,6 +60,25 @@ partitions which data is not written to.
\item{max_partitions}{maximum number of partitions any batch may be
written into. Default is 1024L.}
+\item{max_open_files}{maximum number of files that can be left opened
+during a write operation. If greater than 0 then this will limit the
+maximum number of files that can be left open. If an attempt is made to open
+too many files then the least recently used file will be closed.
+If this setting is set too low you may end up fragmenting your data
+into many small files. The default is 900 which also allows some # of files to be
+open by the scanner before hitting the default Linux limit of 1024.}
+
+\item{max_rows_per_file}{maximum number of rows to be placed in
+any single file}
+
+\item{min_rows_per_group}{write the row groups to the disk when this number of
+rows have accumulated.}
+
+\item{max_rows_per_group}{maximum rows allowed in a single
+group and when this number of rows is exceeded, it is split and the next set
+of rows is written to the next group. This value must be set such that it is
+greater than \code{min_rows_per_group}.}
+
\item{...}{additional format-specific arguments. For available Parquet
options, see \code{\link[=write_parquet]{write_parquet()}}. The available Feather options are:
\itemize{
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#discussion_r794651894
##########
File path: r/R/dataset-write.R
##########
@@ -111,6 +121,10 @@ write_dataset <- function(dataset,
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
Review comment:
Oh that's very nice! Would you mind copying/pasting that into the docs here? IIRC, we can't (easily) link to the C++ docs since the URLs are something like https://arrow.apache.org/docs/cpp/api/dataset.html#_CPPv4N5arrow7dataset29FileSystemDatasetWriteOptions17basename_templateE and change with each version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane closed pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
jonkeane closed pull request #12185:
URL: https://github.com/apache/arrow/pull/12185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033280519
Benchmark runs are scheduled for baseline = 7018a4baf16845d538aee9259000dc1a3ea3f3ab and contender = 0a56006d6184ce9a65056cd7952d168853ecf0b2. 0a56006d6184ce9a65056cd7952d168853ecf0b2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a051f3e283234eeead23c56d8f0e6df5...011061dc9d9748438e7e591c80c37a5e/)
[Finished :arrow_down:0.17% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/cf954642e7ca4941b75dc1f34b6ee437...34302865f4d44128a7a1dd16f5bfbc1d/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/1d37475cb17e47849694f1623da13533...a08c736b334941cb8417609fbff7ad84/)
[Finished :arrow_down:0.17% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/53bb44b13c75410db9687f379297b05d...1ae573c952d948a2818ef35b06750eb0/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033280519
Benchmark runs are scheduled for baseline = 7018a4baf16845d538aee9259000dc1a3ea3f3ab and contender = 0a56006d6184ce9a65056cd7952d168853ecf0b2. 0a56006d6184ce9a65056cd7952d168853ecf0b2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a051f3e283234eeead23c56d8f0e6df5...011061dc9d9748438e7e591c80c37a5e/)
[Finished :arrow_down:0.17% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/cf954642e7ca4941b75dc1f34b6ee437...34302865f4d44128a7a1dd16f5bfbc1d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/1d37475cb17e47849694f1623da13533...a08c736b334941cb8417609fbff7ad84/)
[Finished :arrow_down:0.17% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/53bb44b13c75410db9687f379297b05d...1ae573c952d948a2818ef35b06750eb0/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033227784
@jonkeane I have used `=` instead of `<-`. Thanks for the changes.
Also the test case could get 5 or 6 depending on the how fast batches are read/written?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1018105853
https://issues.apache.org/jira/browse/ARROW-15020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029489818
Hmmm the only build that's failing looks like it's an issue with version numbers. I thought that rebasing on the latest master (which does have the version bump) would solve that, but it seems no?
@thisisnic might have an idea of what we're doing wrong here...
```
:path=>"dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb"},
{:hunks=>
? [["- \"name\": \"8.0 (dev)\",",
? 7
"+ \"name\": \"9.0 (dev)\",",
? "- \"name\": \"7.0 (stable)\",",
? 6
"+ \"name\": \"8.0 (stable)\",",
"+ {",
? "+ \"name\": \"7.0\",",
? 6
? "+ \"version\": \"7.0/\"",
? 6
"+ },"]],
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#discussion_r794606763
##########
File path: r/R/dataset-write.R
##########
@@ -111,6 +121,10 @@ write_dataset <- function(dataset,
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
+ max_rows_per_file = 0L,
+ min_rows_per_group = 0L,
Review comment:
```suggestion
min_rows_per_group = 0L,
```
##########
File path: r/R/dataset-write.R
##########
@@ -50,6 +50,16 @@
#' partitions which data is not written to.
#' @param max_partitions maximum number of partitions any batch may be
#' written into. Default is 1024L.
+#' @param max_open_files maximum number of files that can be left opened
+#' during a write operation. Default is 900L.
+#' @param max_rows_per_file maximum limit how many rows are placed in
+#' any single file
+#' @param min_rows_per_group write the row groups to the disk when sufficient
Review comment:
```suggestion
#' @param min_rows_per_group write the row groups to the disk when this number of
```
##########
File path: r/tests/testthat/test-dataset-write.R
##########
@@ -505,3 +505,145 @@ test_that("Max partitions fails with non-integer values and less than required p
"max_partitions must be a positive, non-missing integer"
)
})
+
+get_num_of_files = function(dir, format) {
+ files <- list.files(dir, pattern = paste('.', format, sep=""),
Review comment:
```suggestion
files <- list.files(dir, pattern = paste(".", format, sep = ""),
```
##########
File path: r/R/dataset-write.R
##########
@@ -50,6 +50,16 @@
#' partitions which data is not written to.
#' @param max_partitions maximum number of partitions any batch may be
#' written into. Default is 1024L.
+#' @param max_open_files maximum number of files that can be left opened
+#' during a write operation. Default is 900L.
+#' @param max_rows_per_file maximum limit how many rows are placed in
+#' any single file
Review comment:
```suggestion
#' @param max_rows_per_file maximum number of rows to be placed in
#' any single file.
```
##########
File path: r/R/dataset-write.R
##########
@@ -146,6 +160,8 @@ write_dataset <- function(dataset,
dataset___Dataset__Write(
options, path_and_fs$fs, path_and_fs$path,
partitioning, basename_template, scanner,
- existing_data_behavior, max_partitions
+ existing_data_behavior, max_partitions,
+ max_open_files, max_rows_per_file,
Review comment:
```suggestion
max_open_files, max_rows_per_file,
```
##########
File path: r/tests/testthat/test-dataset-write.R
##########
@@ -505,3 +505,145 @@ test_that("Max partitions fails with non-integer values and less than required p
"max_partitions must be a positive, non-missing integer"
)
})
+
+get_num_of_files = function(dir, format) {
Review comment:
```suggestion
get_num_of_files <- function(dir, format) {
```
##########
File path: r/R/dataset-write.R
##########
@@ -111,6 +121,10 @@ write_dataset <- function(dataset,
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
Review comment:
Is `900L` here a default in C++? I'm curious where that specific number came from... (it's close to `1024L` or `1000L`, but curious why it's lower?)
##########
File path: r/tests/testthat/test-dataset-write.R
##########
@@ -505,3 +505,145 @@ test_that("Max partitions fails with non-integer values and less than required p
"max_partitions must be a positive, non-missing integer"
)
})
+
+get_num_of_files = function(dir, format) {
+ files <- list.files(dir, pattern = paste('.', format, sep=""),
+ all.files = FALSE, recursive = TRUE, full.names = TRUE)
+ length(files)
+}
+
+test_that("Dataset write max open files", {
+ skip_if_not_available("parquet")
+ # test default partitioning
+ dst_dir <- make_temp_dir()
+ file_format <- "parquet"
+ partitioning <- c("c2")
+ num_of_unique_c2_groups = 5
+
+ record_batch_1 <- record_batch(c1=c(1, 2, 3, 4, 0, 10),
+ c2=c('a', 'b', 'c', 'd', 'e', 'a'))
Review comment:
Also a minor style note but `"` should generally be used over `'`
##########
File path: r/tests/testthat/test-dataset-write.R
##########
@@ -505,3 +505,145 @@ test_that("Max partitions fails with non-integer values and less than required p
"max_partitions must be a positive, non-missing integer"
)
})
+
+get_num_of_files = function(dir, format) {
+ files <- list.files(dir, pattern = paste('.', format, sep=""),
+ all.files = FALSE, recursive = TRUE, full.names = TRUE)
+ length(files)
+}
+
+test_that("Dataset write max open files", {
+ skip_if_not_available("parquet")
+ # test default partitioning
+ dst_dir <- make_temp_dir()
+ file_format <- "parquet"
+ partitioning <- c("c2")
+ num_of_unique_c2_groups = 5
+
+ record_batch_1 <- record_batch(c1=c(1, 2, 3, 4, 0, 10),
Review comment:
This is minor, but typically in R we put spaces around `=`, so `c1=c(1, 2, 3, 4, 0, 10)` should be `c1 = c(1, 2, 3, 4, 0, 10)`
##########
File path: r/R/dataset-write.R
##########
@@ -50,6 +50,16 @@
#' partitions which data is not written to.
#' @param max_partitions maximum number of partitions any batch may be
#' written into. Default is 1024L.
+#' @param max_open_files maximum number of files that can be left opened
+#' during a write operation. Default is 900L.
+#' @param max_rows_per_file maximum limit how many rows are placed in
+#' any single file
+#' @param min_rows_per_group write the row groups to the disk when sufficient
+#' rows have accumulated.
+#' @param max_rows_per_group maximum number of row groups allowed in a single
+#' group and when the rows execeeds, it is splitted and excess is written to
+#' another group. This value must be set such that it is greater than
+#' min_rows_per_group.
Review comment:
```suggestion
#' @param max_rows_per_group maximum number of rows allowed in a single
#' group and when this number of rows is exceeded, it is split and the next set
#' of rows is written to the next group. This value must be set such that it is
#' greater than `min_rows_per_group`.
```
##########
File path: r/R/dataset-write.R
##########
@@ -111,6 +121,10 @@ write_dataset <- function(dataset,
hive_style = TRUE,
existing_data_behavior = c("overwrite", "error", "delete_matching"),
max_partitions = 1024L,
+ max_open_files = 900L,
+ max_rows_per_file = 0L,
Review comment:
It's a bit odd that the default `0L` here actually means no limit. In R I would expect either `NA`, `NULL`, or `Inf` to represent no limit.
For example, when using `write_parquet()` `chunk_size` defaults to `NULL`:
https://github.com/apache/arrow/blob/07ec0a12d430dc9151678b6f00d5c6fc0598f034/r/R/parquet.R#L144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029462418
You're right — it wasn't the linter that was an issue, but there was a function from a previous PR that was causing documentation issues here. I've pushed those docs so that this builds cleanly (and so we didn't conflict or have the builds failing when this is merged — it's a bit unclean in that it's an extra bit of code, but that's ok for this)
I also simplified one of the tests `"Dataset min_rows_per_group"` The for loop + if clause was behaving strangely for me (when I ran it fresh it would always only do the else clause). Regardless, I noticed that the order of which record batch will have the left over row in it isn't deterministic, so we can't check the final element. Instead I'm checking that only one of the row group sizes is less than the min.
I also rebased so that we can get the version bump (which was causing a few of the CI jobs to complain)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vibhatha commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
vibhatha commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1029811567
@kou it resolved the issue, Thank you.
@jonkeane seems everything is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033038247
It looks like one of the tests I edited was slightly non-determenistic (sometimes it would have 5 rowgroups and sometimes 6 since the number of rows divided by 4 and divided by 5 could fit into either. I've made that test fixture slightly smaller so that it always was broken into 3 row groups. This should be more deterministic.
I also rebased to grab some upstream changes (basically the docs changes I added here...) and I'll merge when it's green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12185: ARROW-15020: [R] Add bindings for new dataset writing options
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12185:
URL: https://github.com/apache/arrow/pull/12185#issuecomment-1033280519
Benchmark runs are scheduled for baseline = 7018a4baf16845d538aee9259000dc1a3ea3f3ab and contender = 0a56006d6184ce9a65056cd7952d168853ecf0b2. 0a56006d6184ce9a65056cd7952d168853ecf0b2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a051f3e283234eeead23c56d8f0e6df5...011061dc9d9748438e7e591c80c37a5e/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/cf954642e7ca4941b75dc1f34b6ee437...34302865f4d44128a7a1dd16f5bfbc1d/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/1d37475cb17e47849694f1623da13533...a08c736b334941cb8417609fbff7ad84/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/53bb44b13c75410db9687f379297b05d...1ae573c952d948a2818ef35b06750eb0/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org