You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2021/05/03 13:21:00 UTC
[jira] [Commented] (ARROW-12620) [C++] Dataset writing can only
include projected columns if input columns are also included
[ https://issues.apache.org/jira/browse/ARROW-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338365#comment-17338365 ]
Joris Van den Bossche commented on ARROW-12620:
-----------------------------------------------
Can confirm this in Python:
{code}
import pyarrow.dataset as ds
import pyarrow.parquet as pq
table = pa.table({'a': [1, 2, 3]})
dataset = ds.dataset(table)
projection = {'b': ds.field('a')}
ds.write_dataset(dataset.scanner(columns=projection), "test_scanner.parquet", format="parquet")
In [9]: pq.read_table("test_scanner.parquet").to_pandas()
Out[9]:
b
0 NaN
1 NaN
2 NaN
{code}
> [C++] Dataset writing can only include projected columns if input columns are also included
> -------------------------------------------------------------------------------------------
>
> Key: ARROW-12620
> URL: https://issues.apache.org/jira/browse/ARROW-12620
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 4.0.0
> Reporter: Neal Richardson
> Priority: Major
>
> I discovered this while working on https://github.com/apache/arrow/pull/10191. You can project new columns when writing a dataset, but only if they are derived from columns that are included in the output. Here's an R-based example:
> {code}
> # Simple function to write and re-open the new dataset
> write_then_open <- function(ds, path, ...) {
> write_dataset(ds, path, ...)
> open_dataset(path)
> }
> tab <- Table$create(a = 1:5)
> tab %>%
> write_then_open(ds_dir) %>%
> collect()
> # # A tibble: 5 x 1
> # a
> # <int>
> # 1 1
> # 2 2
> # 3 3
> # 4 4
> # 5 5
> # If you rename a column, it's all nulls
> tab %>%
> select(b = a) %>%
> write_then_open(ds_dir) %>%
> collect()
> # # A tibble: 5 x 1
> # b
> # <int>
> # 1 NA
> # 2 NA
> # 3 NA
> # 4 NA
> # 5 NA
> # If you derive a new column and keep the original, it works
> tab %>%
> mutate(b = a) %>%
> write_then_open(ds_dir) %>%
> collect()
> # # A tibble: 5 x 2
> # a b
> # <int> <int>
> # 1 1 1
> # 2 2 2
> # 3 3 3
> # 4 4 4
> # 5 5 5
> # transmute() only keeps the added columns, so it also illustrates the failure
> tab %>%
> transmute(b = a) %>%
> write_then_open(ds_dir) %>%
> collect()
> # # A tibble: 5 x 1
> # b
> # <int>
> # 1 NA
> # 2 NA
> # 3 NA
> # 4 NA
> # 5 NA
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)