You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "atsuik (via GitHub)" <gi...@apache.org> on 2023/04/14 17:18:54 UTC
[GitHub] [arrow] atsuik opened a new issue, #35140: [R] configure script for R fails to set PKG_CFLAGS
atsuik opened a new issue, #35140:
URL: https://github.com/apache/arrow/issues/35140
### Describe the bug, including details regarding any error messages, version, and platform.
### Describe the bug, including details regarding any error messages, version, and platform.
arrow version: 11.0.0.3
R version: 4.2.3
platform: arm64
I found that current `r/configure` fails to set PKG_CFLAGS parsing ARROW_OPTS_CMAKE file.
This is a log I tried to install arrow to my m1 mac.
```
> install.packages("arrow")
Installing package into ‘/opt/homebrew/lib/R/4.2/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://cran.ism.ac.jp/src/contrib/arrow_11.0.0.3.tar.gz'
Content type 'application/x-gzip' length 3921484 bytes (3.7 MB)
==================================================
downloaded 3.7 MB
* installing *source* package ‘arrow’ ...
files ‘tools/cpp/src/arrow/Testing/Temporary/CTestCostData.txt’, ‘tools/cpp/src/arrow/Testing/Temporary/LastTest.log’ are missing
** using staged installation
*** Arrow C++ libraries found via pkg-config at /opt/homebrew/Cellar/apache-arrow/11.0.0_3/lib
**** Warning: library version mismatch
**** C++ is 11.0.0 but R is 11.0.0.3
**** If installation fails, upgrade the C++ library to match
**** or retry with ARROW_USE_PKG_CONFIG=false
PKG_CFLAGS=
PKG_LIBS=-L/opt/homebrew/Cellar/apache-arrow/11.0.0_3/lib -larrow
```
The log shows `PKG_CFLAGS` is empty.
This causes that capabilities, such as dataset, parquet, etc..., are FALSE.
```
> arrow_info()
Arrow package version: 11.0.0.3
Capabilities:
dataset FALSE
substrait FALSE
parquet FALSE
json FALSE
s3 FALSE
gcs FALSE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc FALSE
Memory:
Allocator jemalloc
Current 0 bytes
Max 0 bytes
Runtime:
SIMD Level none
Detected SIMD Level none
Build:
C++ Library Version 11.0.0
C++ Compiler AppleClang
C++ Compiler Version 14.0.3.14030022
```
This is caused by `$LIB_DIR` is empty at this line
https://github.com/apache/arrow/blob/5c73973a23deb68bef75be8acd743763307ac12f/r/configure#L255
I found that the deletion of line at this pull request
https://github.com/apache/arrow/pull/14235/files#diff-089697faebdb7820ca629a2bb316b878cc0ba18a5bfb0b60996f8dbcd1fa11e7L234
causes this problem.
```
LIB_DIR=`echo $PKG_DIRS | sed -e 's/^-L//'`
```
I have already checked that to recover this line resolves this problem.
This is the log.
```
install.packages("./arrow", repos=NULL, type="source")
Installing package into ‘/opt/homebrew/lib/R/4.2/site-library’
(as ‘lib’ is unspecified)
* installing *source* package ‘arrow’ ...
files ‘tools/cpp/src/arrow/Testing/Temporary/CTestCostData.txt’, ‘tools/cpp/src/arrow/Testing/Temporary/LastTest.log’ are missing
file ‘configure’ has the wrong MD5 checksum
** using staged installation
*** Arrow C++ libraries found via pkg-config at /opt/homebrew/Cellar/apache-arrow/11.0.0_3/lib
**** Warning: library version mismatch
**** C++ is 11.0.0 but R is 11.0.0.3
**** If installation fails, upgrade the C++ library to match
**** or retry with ARROW_USE_PKG_CONFIG=false
PKG_CFLAGS= -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -DARROW_R_WITH_JSON -DARROW_R_WITH_S3
PKG_LIBS=-L/opt/homebrew/Cellar/apache-arrow/11.0.0_3/lib -larrow_dataset -lparquet -larrow
** libs
make: Nothing to be done for `all'.
installing to /opt/homebrew/lib/R/4.2/site-library/00LOCK-arrow/00new/arrow/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (arrow)
> library(arrow)
Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
> arrow_info()
Arrow package version: 11.0.0.3
Capabilities:
dataset TRUE
substrait FALSE
parquet TRUE
json TRUE
s3 TRUE
gcs FALSE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc FALSE
Memory:
Allocator jemalloc
Current 0 bytes
Max 0 bytes
Runtime:
SIMD Level none
Detected SIMD Level none
Build:
C++ Library Version 11.0.0
C++ Compiler AppleClang
C++ Compiler Version 14.0.3.14030022
```
### Component(s)
R
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vkhodygo commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "vkhodygo (via GitHub)" <gi...@apache.org>.
vkhodygo commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1619074260
@nealrichardson just to make this clear: is it included into `12.0.1`? Judging by the timestamps it should be, but the release files do not reflect these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson closed issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson closed issue #35140: [R] configure script for R fails to set PKG_CFLAGS
URL: https://github.com/apache/arrow/issues/35140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1620413790
> @nealrichardson Thanks for the info, do you think you could provide any more accurate ETA for `v13.0.0`?
No, but it will be discussed on the dev@arrow.apache.org mailing list, so you can follow the discussion there.
> Also, is there a temporary fix for this? I could use `13.0.0dev`, but I'm not sure it'll work properly.
I suggested a workaround above, does that work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1509147335
The homebrew package does, but the pkg-config name for that is `arrow-dataset` not `arrow` as we test for:
```
ls /opt/homebrew/Cellar/apache-arrow/11.0.0_2/lib/pkgconfig/
arrow-compute.pc arrow-flight-sql.pc arrow.pc
arrow-csv.pc arrow-flight.pc gandiva.pc
arrow-dataset.pc arrow-json.pc parquet.pc
arrow-filesystem.pc arrow-orc.pc plasma.pc
```
That's a problem wherever a system install of Arrow exists (e.g., https://github.com/apache/arrow/issues/31989 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vkhodygo commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "vkhodygo (via GitHub)" <gi...@apache.org>.
vkhodygo commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1643810791
@nealrichardson
> I suggested a workaround above, does that work for you?
Apologies, I decided to put this on hold and wait for the official release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1509125816
> I found that the deletion of line at this pull request
> https://github.com/apache/arrow/pull/14235/files#diff-089697faebdb7820ca629a2bb316b878cc0ba18a5bfb0b60996f8dbcd1fa11e7L234
> causes this problem.
If I recall, the point of the change you identified was that we're using the generated cmake configuration to understand the features of the C++ library. So I would guess then that the Homebrew package isn't including these cmake files, and it probably should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1522232002
In https://github.com/conda-forge/r-arrow-feedstock/pull/63 we found a workaround for this for 11.0.0: in your case, set the env var `LIB_DIR=/opt/homebrew/Cellar/apache-arrow/11.0.0_3/lib` and it should find the features correctly. It turns out this has already been fixed in `main`, so it will no longer be an issue in the upcoming release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] paleolimbot commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1509111363
I wonder if the root of your problem is that you have Arrow installed via homebrew which is getting picked up by pkg-config
Is there a reason you need to install the package from source? The binary install from CRAN is the preferred way to install on MacOS since it already contains all the components noted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1518773354
https://github.com/conda-forge/r-arrow-feedstock/issues/56#issuecomment-1518745545 may be another manifestation of this (or may not, I haven't investigated)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1619084677
@vkhodygo no it is not, the PR that resolved this, #35147, was a significant rewrite and not a mere bugfix, so it was not included in the 12.0.1 patch release. But 13.0.0 should be coming in the next couple of weeks, and it will be in that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1522555968
FYI: We can use `LIB_DIR=$(brew --prefix apache-arrow)/lib` instead of `LIB_DIR=/opt/homebrew/Cellar/apache-arrow/11.0.0_3/lib`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1522570652
Oh yes, of course. I wasn't trying to suggest a robust solution since this will be no longer useful after the 12.0.0 release closes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vkhodygo commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "vkhodygo (via GitHub)" <gi...@apache.org>.
vkhodygo commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1620178679
@nealrichardson Thanks for the info, do you think you could provide any more accurate ETA for `v13.0.0`? Also, is there a temporary fix for this? I could use `13.0.0dev`, but I'm not sure it'll work properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on issue #35140: [R] configure script for R fails to set PKG_CFLAGS
Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on issue #35140:
URL: https://github.com/apache/arrow/issues/35140#issuecomment-1509271404
But this is about cmake and not pkg-config. It turns out the cmake files are present in the homebrew build:
```
% ls /usr/local/Cellar/apache-arrow/11.0.0_3/lib/cmake/Arrow
ArrowConfig.cmake FindBrotliAlt.cmake Findlz4Alt.cmake
ArrowConfigVersion.cmake FindOpenSSLAlt.cmake Findre2Alt.cmake
ArrowOptions.cmake FindProtobufAlt.cmake Findutf8proc.cmake
ArrowTargets-release.cmake FindSnappyAlt.cmake FindzstdAlt.cmake
ArrowTargets.cmake FindThriftAlt.cmake arrow-config.cmake
FindAWSSDKAlt.cmake FindgRPCAlt.cmake
```
I misread the original report, it's about `LIB_DIR` not being set in the case where the libs are found by pkg-config, and that's why ArrowOptions.cmake isn't being found. I think restoring the deleted line they mentioned, probably protected inside `if [ "$LIB_DIR" = "" ];` for the other cases where `LIB_DIR` does get set correctly, would fix this (and possibly #31989?).
`arrow` vs. `arrow-dataset` isn't relevant here because all of the option flags for all of the libs are in `lib/cmake/Arrow/ArrowOptions.cmake`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org