You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "JosiahParry (via GitHub)" <gi...@apache.org> on 2023/09/25 15:24:12 UTC
[GitHub] [arrow] JosiahParry opened a new issue, #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow
JosiahParry opened a new issue, #37860:
URL: https://github.com/apache/arrow/issues/37860
### Describe the bug, including details regarding any error messages, version, and platform.
I can add metadata to a parquet dataset directly in R like the below. (As an aside, the `WithMetadata()` R6 method does not assign metadata, that can be cleared up / documented better :P)
However, reading the resultant dataset from PyArrow cannot find the metadata.
```r
library(arrow)
tbl <- arrow_table(iris)
cur_meta <- tbl$metadata
cur_meta[["test"]] <- "testing other metadata"
tbl$metadata <- cur_meta
write_dataset(tbl, "data/iris", partitioning = "Species")
open_dataset("data/iris")$metadata
#> $r
#> $r$attributes
#> $r$attributes$class
#> [1] "data.frame"
#>
#> $r$columns
#> $r$columns$Sepal.Length
#> NULL
#>
#> $r$columns$Sepal.Width
#> NULL
#>
#> $r$columns$Petal.Length
#> NULL
#>
#> $r$columns$Petal.Width
#> NULL
#>
#> $r$columns$Species
#> NULL
#>
#> $test
#> [1] "testing other metadata"
```
The result of the pyarrow call is here:
```python
import pyarrow.parquet as pq
pq.ParquetDataset("data/iris").metadata
#> AttributeError: '_ParquetDatasetV2' object has no attribute 'metadata'
```
Metadata should persist / be available by other languages reading the files
### Component(s)
Parquet, Python, R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] JosiahParry commented on issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow
Posted by "JosiahParry (via GitHub)" <gi...@apache.org>.
JosiahParry commented on issue #37860:
URL: https://github.com/apache/arrow/issues/37860#issuecomment-1734428683
Ah, okay. That works!
```python
import pyarrow.parquet as pq
pq.ParquetDataset("data/iris").schema.metadata[b'weighting']
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot commented on issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #37860:
URL: https://github.com/apache/arrow/issues/37860#issuecomment-1734359101
I think the arrow R package implements the `$metadata` shortcut for `Table` and `Dataset` objects, whereas pyarrow does not. I think the metadata is there, it just needs an extra `.schema`:
``` r
library(arrow, warn.conflicts = FALSE)
tf <- tempfile()
tbl <- arrow_table(iris)
cur_meta <- tbl$metadata
cur_meta[["test"]] <- "testing other metadata"
tbl$metadata <- cur_meta
write_dataset(tbl, tf, partitioning = "Species")
open_dataset(tf)$schema$metadata$test
#> [1] "testing other metadata"
pa_dataset <- reticulate::import("pyarrow.dataset", convert = FALSE)
ds <- pa_dataset$dataset(tf)
ds$schema$metadata$keys()
#> dict_keys([b'r', b'test'])
```
Or in regular Python land:)
```python
from pyarrow import dataset
ds = dataset.dataset("path/to/dataset")
ds.schema.metadata[b"test"]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] JosiahParry closed issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow
Posted by "JosiahParry (via GitHub)" <gi...@apache.org>.
JosiahParry closed issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow
URL: https://github.com/apache/arrow/issues/37860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org