You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "JosiahParry (via GitHub)" <gi...@apache.org> on 2023/09/25 15:24:12 UTC

[GitHub] [arrow] JosiahParry opened a new issue, #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow

JosiahParry opened a new issue, #37860:
URL: https://github.com/apache/arrow/issues/37860

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I can add metadata to a parquet dataset directly in R like the below. (As an aside, the `WithMetadata()` R6 method does not assign metadata, that can be cleared up / documented better :P) 
   
   However, reading the resultant dataset from PyArrow cannot find the metadata. 
   
   ```r
   library(arrow)
   tbl <- arrow_table(iris)
   
   cur_meta <- tbl$metadata
   cur_meta[["test"]] <- "testing other metadata"
   tbl$metadata <- cur_meta
   
   write_dataset(tbl, "data/iris", partitioning = "Species")
   open_dataset("data/iris")$metadata
   #> $r
   #> $r$attributes
   #> $r$attributes$class
   #> [1] "data.frame"
   #> 
   #> $r$columns
   #> $r$columns$Sepal.Length
   #> NULL
   #> 
   #> $r$columns$Sepal.Width
   #> NULL
   #> 
   #> $r$columns$Petal.Length
   #> NULL
   #> 
   #> $r$columns$Petal.Width
   #> NULL
   #> 
   #> $r$columns$Species
   #> NULL
   #> 
   #> $test
   #> [1] "testing other metadata"
   ```
   
   The result of the pyarrow call is here: 
   
   ```python
   import pyarrow.parquet as pq
   pq.ParquetDataset("data/iris").metadata
   #> AttributeError: '_ParquetDatasetV2' object has no attribute 'metadata'
   ```
   
   Metadata should persist / be available by other languages reading the files
   
   ### Component(s)
   
   Parquet, Python, R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] JosiahParry commented on issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow

Posted by "JosiahParry (via GitHub)" <gi...@apache.org>.
JosiahParry commented on issue #37860:
URL: https://github.com/apache/arrow/issues/37860#issuecomment-1734428683

   Ah,  okay. That works! 
   
   ```python
   import pyarrow.parquet as pq
   pq.ParquetDataset("data/iris").schema.metadata[b'weighting']
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #37860:
URL: https://github.com/apache/arrow/issues/37860#issuecomment-1734359101

   I think the arrow R package implements the `$metadata` shortcut for `Table` and `Dataset` objects, whereas pyarrow does not. I think the metadata is there, it just needs an extra `.schema`:
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   
   tf <- tempfile()
   tbl <- arrow_table(iris)
   
   cur_meta <- tbl$metadata
   cur_meta[["test"]] <- "testing other metadata"
   tbl$metadata <- cur_meta
   
   write_dataset(tbl, tf, partitioning = "Species")
   open_dataset(tf)$schema$metadata$test
   #> [1] "testing other metadata"
   
   pa_dataset <- reticulate::import("pyarrow.dataset", convert = FALSE)
   ds <- pa_dataset$dataset(tf)
   ds$schema$metadata$keys()
   #> dict_keys([b'r', b'test'])
   ```
   
   Or in regular Python land:)
   
   ```python
   from pyarrow import dataset
   
   ds = dataset.dataset("path/to/dataset")
   ds.schema.metadata[b"test"]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] JosiahParry closed issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow

Posted by "JosiahParry (via GitHub)" <gi...@apache.org>.
JosiahParry closed issue #37860: [R][arrow] write_dataset() parquet metadata not available in PyArrow
URL: https://github.com/apache/arrow/issues/37860


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org