You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/10 18:56:39 UTC
[GitHub] [arrow] nealrichardson opened a new pull request, #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
nealrichardson opened a new pull request, #13563:
URL: https://github.com/apache/arrow/pull/13563
See reprex (sans terminal formatting) in [r/tests/testthat/_snaps/dplyr-glimpse.md](https://github.com/apache/arrow/compare/master...nealrichardson:arrow:glimpse-part1?expand=1#diff-e8d50da600908f077796a43b7600c17d34448671c7975bb8c4056a484ac2999e)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918118858
##########
r/R/dplyr.R:
##########
@@ -276,13 +278,48 @@ source_data <- function(x) {
}
}
-is_collapsed <- function(x) inherits(x$.data, "arrow_dplyr_query")
+all_sources <- function(x) {
+ if (is.null(x)) {
+ x
+ } else if (!inherits(x, "arrow_dplyr_query")) {
+ list(x)
+ } else {
+ c(
+ all_sources(x$.data),
+ all_sources(x$join$right_data),
+ all_sources(x$union_all$right_data)
+ )
+ }
+}
-has_aggregation <- function(x) {
- # TODO: update with joins (check right side data too)
- !is.null(x$aggregations) || (is_collapsed(x) && has_aggregation(x$.data))
+query_can_stream <- function(x) {
Review Comment:
> We could build an ExecPlan, but it wouldn't tell us anything about how it would perform, would it?
I'm not super close to the ExecPlan code, but I thought they were composed of a graph of nodes that could be traversed and analyzed, just like our `arrow_dplyr_query` structure. Am I wrong on that?
> I'm trying to detect cases where I can just take head() of the data without having to scan an entire dataset.
I was just thinking that having such a method on `ExecPlan` would be useful in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r917516817
##########
r/R/dplyr-count.R:
##########
@@ -39,7 +39,7 @@ count.Dataset <- count.ArrowTabular <- count.RecordBatchReader <- count.arrow_dp
#' @importFrom rlang sym :=
tally.arrow_dplyr_query <- function(x, wt = NULL, sort = FALSE, name = NULL) {
- check_name <- utils::getFromNamespace("check_name", "dplyr")
+ check_name <- getFromNamespace("check_name", "dplyr")
Review Comment:
Question: Is there a reason we didn't want the `utils::`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13563:
URL: https://github.com/apache/arrow/pull/13563#issuecomment-1179780558
https://issues.apache.org/jira/browse/ARROW-16776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918069720
##########
r/R/dplyr-glimpse.R:
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#' @importFrom utils getFromNamespace
+glimpse.ArrowTabular <- function(x,
+ width = getOption("pillar.width", getOption("width")),
+ ...) {
+ # This function is inspired by pillar:::glimpse.tbl(), with some adaptations
+
+ # We use cli:: and pillar:: throughout this function. We don't need to check
+ # to see if they're installed because dplyr depends on pillar, which depends
+ # on cli, and we're only in this function though S3 dispatch on dplyr::glimpse
+ if (!is.finite(width)) {
+ abort("`width` must be finite.")
+ }
+
+ # We need a couple of internal functions in pillar for formatting
+ pretty_int <- getFromNamespace("big_mark", "pillar")
+ make_shorter <- getFromNamespace("str_trunc", "pillar")
+ tickify <- getFromNamespace("tick_if_needed", "pillar")
+
+ # Even though this is the ArrowTabular method, we use it for arrow_dplyr_query
+ # so make some accommodations. (Others are handled by S3 method dispatch.)
+ if (inherits(x, "arrow_dplyr_query")) {
+ class_title <- paste(source_data(x)$class_title(), "(query)")
+ } else {
+ class_title <- x$class_title()
+ }
+ cli::cat_line(class_title)
+
+ dims <- dim(x)
+ cli::cat_line(sprintf(
+ "%s rows x %s columns", pretty_int(dims[1]), pretty_int(dims[2])
+ ))
+
+ if (dims[2] == 0) {
+ return(invisible(x))
+ }
+
+ nrows <- as.integer(width / 3)
+ head_tab <- dplyr::compute(head(x, nrows))
+ # Take the schema from this Table because if x is arrow_dplyr_query, some
+ # output types could be a best guess (in implicit_schema()).
+ schema <- head_tab$schema
+
+ # Assemble the column names and types
+ # We use the Arrow type names here. See type_sum.DataType() below.
+ var_types <- map_chr(schema$fields, ~ format(pillar::new_pillar_type(.$type)))
+ # glimpse.tbl() left-aligns the var names (pads with whitespace to the right)
+ # and appends the types next to them. Because those type names are
+ # aggressively truncated to all be roughly the same length, this means the
+ # data glimpse that follows is also mostly aligned.
+ # However, Arrow type names are longer and variable length, and we're only
+ # truncating the nested type information inside of <...>. So, to keep the
+ # data glimpses aligned, we "justify" align the name and type: add the padding
+ # whitespace between them so that the total width is equal.
+ var_headings <- paste("$", center_pad(tickify(names(x)), var_types))
+
+ # Assemble the data glimpse
+ df <- as.data.frame(head_tab)
+ formatted_data <- map_chr(df, function(.) {
+ tryCatch(
+ paste(pillar::format_glimpse(.), collapse = ", "),
+ # This could error e.g. if you have a VctrsExtensionType and the package
+ # that defines methods for the data is not loaded
+ error = function(e) conditionMessage(e)
+ )
+ })
+ # Here and elsewhere in the glimpse code, you have to use pillar::get_extent()
+ # instead of nchar() because get_extent knows how to deal with ANSI escapes
+ # etc.--it counts how much space on the terminal will be taken when printed.
+ data_width <- width - pillar::get_extent(var_headings)
+ truncated_data <- make_shorter(formatted_data, data_width)
+
+ # Print the table body (var name, type, data glimpse)
+ cli::cat_line(var_headings, " ", truncated_data)
+
+ # TODO: use crayon to style these footers?
+ if (inherits(x, "arrow_dplyr_query")) {
+ cli::cat_line("Call `print()` for query details")
+ } else if (any(grepl("<...>", var_types, fixed = TRUE)) || schema$HasMetadata) {
+ cli::cat_line("Call `print()` for full schema details")
+ }
+ invisible(x)
+}
+
+# Dataset has an efficient head() method via Scanner so this is fine
+glimpse.Dataset <- glimpse.ArrowTabular
+
+glimpse.arrow_dplyr_query <- function(x,
+ width = getOption("pillar.width", getOption("width")),
+ ...) {
+ if (any(map_lgl(all_sources(x), ~ inherits(., "RecordBatchReader")))) {
+ msg <- paste(
+ "Cannot glimpse() data from a RecordBatchReader because it can only be",
+ "read one time. Call `compute()` to evaluate the query first."
+ )
+ message(msg)
+ print(x)
+ } else if (query_on_dataset(x) && !query_can_stream(x)) {
+ msg <- paste(
+ "This query requires a full table scan, so glimpse() may be",
+ "expensive. Call `compute()` to evaluate the query first."
+ )
+ message(msg)
+ print(x)
+ } else {
+ # Go for it
+ glimpse.ArrowTabular(x, width = width, ...)
+ }
+}
+
+glimpse.RecordBatchReader <- function(x,
+ width = getOption("pillar.width", getOption("width")),
+ ...) {
+ # TODO(ARROW-17038): to_arrow() on duckdb con should hold con not RBR so it
+ # can be run more than once (like duckdb does on the other side)
+ msg <- paste(
+ "Cannot glimpse() data from a RecordBatchReader because it can only be",
+ "read one time; call `as_arrow_table()` to consume it first"
Review Comment:
```suggestion
"read one time; call `as_arrow_table()` to consume it first."
```
##########
r/R/dplyr.R:
##########
@@ -276,13 +278,48 @@ source_data <- function(x) {
}
}
-is_collapsed <- function(x) inherits(x$.data, "arrow_dplyr_query")
+all_sources <- function(x) {
+ if (is.null(x)) {
+ x
+ } else if (!inherits(x, "arrow_dplyr_query")) {
+ list(x)
+ } else {
+ c(
+ all_sources(x$.data),
+ all_sources(x$join$right_data),
+ all_sources(x$union_all$right_data)
+ )
+ }
+}
-has_aggregation <- function(x) {
- # TODO: update with joins (check right side data too)
- !is.null(x$aggregations) || (is_collapsed(x) && has_aggregation(x$.data))
+query_can_stream <- function(x) {
Review Comment:
The reason we can't push this down to C++ is because we haven't constructed an exec plan yet, right? Otherwise, it would be more maintainable to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jthomasmock commented on pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
jthomasmock commented on PR #13563:
URL: https://github.com/apache/arrow/pull/13563#issuecomment-1180540635
> @jthomasmock FYI
Very excited to test this out once it's in dev, had peeked through the `testthat` outputs/snapshots!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
westonpace commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918160386
##########
r/R/dplyr.R:
##########
@@ -276,13 +278,48 @@ source_data <- function(x) {
}
}
-is_collapsed <- function(x) inherits(x$.data, "arrow_dplyr_query")
+all_sources <- function(x) {
+ if (is.null(x)) {
+ x
+ } else if (!inherits(x, "arrow_dplyr_query")) {
+ list(x)
+ } else {
+ c(
+ all_sources(x$.data),
+ all_sources(x$join$right_data),
+ all_sources(x$union_all$right_data)
+ )
+ }
+}
-has_aggregation <- function(x) {
- # TODO: update with joins (check right side data too)
- !is.null(x$aggregations) || (is_collapsed(x) && has_aggregation(x$.data))
+query_can_stream <- function(x) {
Review Comment:
> I was just thinking that having such a method on ExecPlan would be useful in general.
Possibly. We'd probably want to define it more formally. SQL has `LIMIT X` and Substrait's equivalent is [`FetchRel`](https://substrait.io/relations/logical_relations/#fetch-operation). Neither of these are exactly what is being detected here. For example, it is legal to have `SELECT SUM(x) FROM table LIMIT 1` but it wouldn't actually limit any data being read.
We could define it as "single pipeline queries" but a pipeline breaker doesn't necessarily mean a query is non-streaming (for example, hash-join is sometimes permitted as "streaming" in this example but it is always a pipeline breaker).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918202063
##########
r/R/dplyr.R:
##########
@@ -276,13 +278,48 @@ source_data <- function(x) {
}
}
-is_collapsed <- function(x) inherits(x$.data, "arrow_dplyr_query")
+all_sources <- function(x) {
+ if (is.null(x)) {
+ x
+ } else if (!inherits(x, "arrow_dplyr_query")) {
+ list(x)
+ } else {
+ c(
+ all_sources(x$.data),
+ all_sources(x$join$right_data),
+ all_sources(x$union_all$right_data)
+ )
+ }
+}
-has_aggregation <- function(x) {
- # TODO: update with joins (check right side data too)
- !is.null(x$aggregations) || (is_collapsed(x) && has_aggregation(x$.data))
+query_can_stream <- function(x) {
Review Comment:
Since you mentioned `limit`, I'll make a plug for ARROW-16628. Not relevant for this particular question, just would let me delete some R specific handling outside of the ExecPlan, and I'm guessing we'll have to do it to support substrait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13563:
URL: https://github.com/apache/arrow/pull/13563#issuecomment-1180439018
@jthomasmock FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r917848669
##########
r/R/dplyr-count.R:
##########
@@ -39,7 +39,7 @@ count.Dataset <- count.ArrowTabular <- count.RecordBatchReader <- count.arrow_dp
#' @importFrom rlang sym :=
tally.arrow_dplyr_query <- function(x, wt = NULL, sort = FALSE, name = NULL) {
- check_name <- utils::getFromNamespace("check_name", "dplyr")
+ check_name <- getFromNamespace("check_name", "dplyr")
Review Comment:
We've elsewhere done importFrom so it's not necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918129935
##########
r/R/dplyr.R:
##########
@@ -276,13 +278,48 @@ source_data <- function(x) {
}
}
-is_collapsed <- function(x) inherits(x$.data, "arrow_dplyr_query")
+all_sources <- function(x) {
+ if (is.null(x)) {
+ x
+ } else if (!inherits(x, "arrow_dplyr_query")) {
+ list(x)
+ } else {
+ c(
+ all_sources(x$.data),
+ all_sources(x$join$right_data),
+ all_sources(x$union_all$right_data)
+ )
+ }
+}
-has_aggregation <- function(x) {
- # TODO: update with joins (check right side data too)
- !is.null(x$aggregations) || (is_collapsed(x) && has_aggregation(x$.data))
+query_can_stream <- function(x) {
Review Comment:
Sure, that probably would be useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918406253
##########
r/R/chunked-array.R:
##########
@@ -119,6 +119,7 @@ ChunkedArray <- R6Class("ChunkedArray",
assert_that(!is.na(descending))
# TODO: after ARROW-12042 is closed, review whether this and the
# Array$SortIndices definition can be consolidated
+ # (edit: ARROW-12042 was resolved in 7.0.0)
Review Comment:
Dealt with this in https://github.com/apache/arrow/pull/13576
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson merged pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson merged PR #13563:
URL: https://github.com/apache/arrow/pull/13563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918203178
##########
r/tests/testthat/_snaps/dplyr-glimpse.md:
##########
@@ -0,0 +1,152 @@
+# glimpse() Table/ChunkedArray
+
+ Code
+ glimpse(tab)
+ Output
+ Table
+ 10 rows x 7 columns
+ $ int <int32> 1, 2, 3, NA, 5, 6, 7, 8, 9, 10
+ $ dbl <double> 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, NA, 10.1
+ $ dbl2 <double> 5, 5, 5, 5, 5, 5, 5, 5, 5, 5
+ $ lgl <bool> TRUE, NA, TRUE, FALSE, TRUE, NA, NA, FALSE, FALSE, NA
+ $ false <bool> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ~
+ $ chr <string> "a", "b", "c", "d", "e", NA, "g", "h", "i", "j"
+ $ fct <dictionary<...>> a, b, c, d, NA, NA, g, h, i, j
+ Call `print()` for full schema details
+
+---
+
+ Code
+ glimpse(tab$chr)
+ Output
+ <string> [ [ "a", "b", "c", "d", "e", null, "g", "h", "i", "j" ] ]
+
+# glimpse() RecordBatch/Array
+
+ Code
+ glimpse(batch)
+ Output
+ RecordBatch
+ 10 rows x 7 columns
+ $ int <int32> 1, 2, 3, NA, 5, 6, 7, 8, 9, 10
+ $ dbl <double> 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, NA, 10.1
+ $ dbl2 <double> 5, 5, 5, 5, 5, 5, 5, 5, 5, 5
+ $ lgl <bool> TRUE, NA, TRUE, FALSE, TRUE, NA, NA, FALSE, FALSE, NA
+ $ false <bool> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, ~
+ $ chr <string> "a", "b", "c", "d", "e", NA, "g", "h", "i", "j"
+ $ fct <dictionary<...>> a, b, c, d, NA, NA, g, h, i, j
+ Call `print()` for full schema details
+
+---
+
+ Code
+ glimpse(batch$int)
+ Output
+ <int32> [ 1, 2, 3, null, 5, 6, 7, 8, 9, 10 ]
+
+# glimpse() with VctrsExtensionType
+
+ Code
+ glimpse(haven)
+ Output
+ Table
+ 2 rows x 3 columns
+ $ num <double> 5.1, 4.9
+ $ cat_int <ext<hvn_lbll>> 3, 1
+ $ cat_chr <ext<hvn_lbll>> Can't convert `x` <haven_labelled> to <character>.
+ Call `print()` for full schema details
+
+---
+
+ Code
+ glimpse(haven[[3]])
+ Output
+ <<haven_labelled[0]>> [ [ "B", "B" ] ]
+
+# glimpse prints message about schema if there are complex types
+
+ Code
+ glimpse(dictionary_but_no_metadata)
+ Output
+ Table
+ 5 rows x 2 columns
+ $ a <int32> 1, 2, 3, 4, 5
+ $ b <dictionary<...>> 1, 2, 3, 4, 5
+ Call `print()` for full schema details
+
+---
+
+ Code
+ glimpse(Table$create(a = 1))
+ Output
+ Table
+ 1 rows x 1 columns
+ $ a <double> 1
+
+# glimpse() calls print() instead of showing data for RBR
+
+ Code
+ example_data %>% as_record_batch_reader() %>% glimpse()
+ Message <simpleMessage>
+ Cannot glimpse() data from a RecordBatchReader because it can only be read one time; call `as_arrow_table()` to consume it first
Review Comment:
Fix to match the previous punctuation change.
```suggestion
Cannot glimpse() data from a RecordBatchReader because it can only be read one time; call `as_arrow_table()` to consume it first.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a diff in pull request #13563: ARROW-16776: [R] dplyr::glimpse method for arrow table and datasets
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13563:
URL: https://github.com/apache/arrow/pull/13563#discussion_r918105901
##########
r/R/dplyr.R:
##########
@@ -276,13 +278,48 @@ source_data <- function(x) {
}
}
-is_collapsed <- function(x) inherits(x$.data, "arrow_dplyr_query")
+all_sources <- function(x) {
+ if (is.null(x)) {
+ x
+ } else if (!inherits(x, "arrow_dplyr_query")) {
+ list(x)
+ } else {
+ c(
+ all_sources(x$.data),
+ all_sources(x$join$right_data),
+ all_sources(x$union_all$right_data)
+ )
+ }
+}
-has_aggregation <- function(x) {
- # TODO: update with joins (check right side data too)
- !is.null(x$aggregations) || (is_collapsed(x) && has_aggregation(x$.data))
+query_can_stream <- function(x) {
Review Comment:
I don't follow. We could build an ExecPlan, but it wouldn't tell us anything about how it would perform, would it? I'm trying to detect cases where I can just take `head()` of the data without having to scan an entire dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org