[jira] [Commented] (ARROW-6960) Add information about zstd/lz4 codec installation and linkages for R users

["Neal Richardson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956483#comment-16956483 ] 

Neal Richardson commented on ARROW-6960:
----------------------------------------

It is not possible to add them after the fact. You need the Arrow C++ library built with those codecs, and then you'd need to install the R package from source, linking to that library. On other platforms, I could point you at a few options for doing that, but not Windows. 

On Windows, we could try to support more codecs in the official build. If you wanted to submit a PR to do that, you'd want to modify https://github.com/apache/arrow/blob/master/ci/PKGBUILD (a copy of https://github.com/r-windows/rtools-packages/blob/master/mingw-w64-arrow/PKGBUILD, which feeds into what CRAN uses to build its binaries) to include additional dependencies and turn the right cmake flags on, which you identified. https://github.com/r-windows/rtools-packages shows what libraries should be available already--I see lz4 but not zstd or brotli. So lz4 might be trivial to add. For the others, you'd need to first add it to https://github.com/r-windows/rtools-packages/; its README has a brief note at the bottom for how to approach it. 

> Add information about zstd/lz4 codec installation and linkages for R users
> --------------------------------------------------------------------------
>
>                 Key: ARROW-6960
>                 URL: https://issues.apache.org/jira/browse/ARROW-6960
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: R
>    Affects Versions: 0.15.0
>         Environment: Windows 10
>            Reporter: Grant Nguyen
>            Priority: Minor
>
> When I attempt to write a parquet file using lz4, zstd, or brotli compression using R arrow 0.15.0, I am unable to do so due to the codec support not being built (example below).
>  
> {code:java}
> > arrow::write_parquet(payout_strategy, sink = "records_test_lz4.parquet",compression = "lz4")
> Error in parquet___arrow___FileWriter__WriteTable(self, table, chunk_size) : 
>  Arrow error: IOError: Arrow error: NotImplemented: LZ4 codec support not built{code}
>  
> I believe that the error is generated through [https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/compression.cc#L124-L145], but I am not sure how to call 
> {code:java}
> install.packages("arrow"){code}
> in R to enable the ARROW_WITH_ZSTD/LZ4/BROTLI flags, or whether I should be doing installing zstd separately from arrow and then doing something pre- or post-install to link zstd with arrow. From [https://github.com/apache/arrow/issues/1209], it appears that zstd support has been added to arrow and parquet in general, and the R package readme ([https://github.com/apache/arrow/tree/master/r)|https://github.com/apache/arrow/tree/master/r] notes "On macOS and Windows, installing a binary package from CRAN will handle Arrow's C++ dependencies for you", but I get the sense that does not apply to zstd.
>  
> Is there guidance as to how to enable zstd and other compression codecs prior to or after downloading the R arrow package? Could this be added to the R documentation somewhere for future reference?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)