[jira] [Created] (ARROW-5190) Discussion: tibble dependency in R package

["James Lamb (JIRA)" <ji...@apache.org>]

James Lamb created ARROW-5190:
---------------------------------

             Summary: Discussion: tibble dependency in R package
                 Key: ARROW-5190
                 URL: https://issues.apache.org/jira/browse/ARROW-5190
             Project: Apache Arrow
          Issue Type: Wish
          Components: R
            Reporter: James Lamb


Hello,

 

I would like to have a discussion on the use of *tibble* in the Apache Arrow R package. I looked at the [the project contributor guidelines|[https://github.com/apache/arrow/blob/master/docs/source/developers/contributing.rst]] and could not tell where the best place might be to start a public discussion on this topic, so I decided on JIRA. I apologize if this is not the right place.

 

*TL;DR*

I would like to propose moving the *tibble* dependency in the *arrow* R package to "Suggests", removing the _as_tibble()_ in _read_arrow()_, and having the core R code implementing the Arrow API only return data.frames or other base-R data structures wherever possible.

 

*Reasoning*

[As far as I can tell|[https://github.com/apache/arrow/search?p=1&q=tibble&unscoped_q=tibble]], outside of tests and examples *tibble* is only used in three places in the package:
 * S3 methods to convert Arrow objects to tibbles (_as_tibble.arrow__::__RecordBatch()_, _as.tibble.arrow::Table()_)
 * optional "convert to tibble on the way out" behavior controlled by a flag in interfaces to file types (parquet and feather)
 * [_read_arrow()_|[https://github.com/apache/arrow/blob/0536ef8174982a7a13a251174cc38701e8663b68/r/R/read_table.R#L88]]

 

In my opinion, all three of these uses of *tibble* are valuable for developers who use that package (or other packages in its ecosystem), but I am not convinced that the Arrow R package should be tightly coupled to them or that Arrow maintainers should have to maintain them.

In the Python community, *pandas* is a broadly agreed-upon standard for representing data frames. Even with that ubiquity, *pyarrow* does not depend on *pandas* (it is not necessary to work with it) and all "compatibility with *pandas*" code is isolated in a place explicitly intended for that purpose: [https://github.com/apache/arrow/blob/master/python/pyarrow/pandas_compat.py]

I think that is the ideal handling for integration of Arrow extensions with other software it might be used with. This allows users who care about only one of the integrations (e.g. feather, parquet, HDFS, Apache Spark, tibble, data.table, etc.) to only have to build things they're already using. 

 

*Other background information*

I took the time to write this tonight after talking a colleague through the issues *feather* (R package) users experienced after the *tibble 2.0* release. See for example [wesm/feather#374|[https://github.com/wesm/feather/issues/374]] and [wesm/feather#372|[https://github.com/wesm/feather/issues/37|https://github.com/wesm/feather/issues/374]2]. When *tibble 2.0* came out it broke *feather 0.3.1* and the maintainers there promptly released to CRAN a *feather 0.3.2* which was compatible with *tibble 2.0+*. Unfortunately, this still caused disruptions for many people using *feather* (who inadvertently had *tibble* upgraded as part of installing other packages which depended on it). Nothing about *tibble* was necessary to the implementation of _read_feather()_, as far as I can tell, but this design choice made installing and upgrading *tibble* non-optional for developers who just wanted to use the feather file format and all it's awesome features.

 

If the proposal here is accepted, I hope it will mean we can prevent repeating the same experience with the R *arrow* package and set a strong precedent for developers who want to add compatibility in this package for other members of the ecosystem like parquet or Apache Spark.

 

 

Thank you for hearing me out!

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)