You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "aquasync (via GitHub)" <gi...@apache.org> on 2024/02/14 00:23:06 UTC
[I] Unable to load pyarrow package through reticulate in R on windows [arrow]
aquasync opened a new issue, #40073:
URL: https://github.com/apache/arrow/issues/40073
### Describe the bug, including details regarding any error messages, version, and platform.
I'm trying to use arrow to share large data frames between python and R. If I load the R packages then try to import the python module this fails with an ImportError from python:
```
> library(arrow)
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
> library(reticulate)
> py_discover_config()
...
version: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:29:04) [MSC v.1929 64 bit (AMD64)]
Architecture: 64bit
...
> pa <- import("pyarrow")
Error in py_module_import(module, convert = convert) :
ImportError: DLL load failed while importing lib: The specified procedure could not be found.
Run `reticulate::py_last_error()` for details.
```
Note that I can load the pyarrow in a vanilla python REPL just fine.
If I ensure to import pyarrow before loading the R package then it works, however this is difficult to guarantee (other packages may have already loaded the R arrow package), and causes failures during package development.
```
> library(reticulate)
> import('pyarrow')
Module(pyarrow)
> library(arrow)
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
```
Because it seems to be driven by some sort of incompatibility between the R and python arrow libraries, I tried various different versions to see if I could get anything that works. Initially I was using R 4.2.1, but could also reproduce with a fresh install of R 4.3.2. I tried various python versions including fresh installs of the latest (3.11.5), and also the conda version as installed by `reticulate::install_miniconda` (3.9.18 above). I tried using the latest version of the pyarrow package (15.0.0), and also downgrading to 14.0.0 to match the latest available R package.
### Component(s)
Python, R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "annakrystalli (via GitHub)" <gi...@apache.org>.
annakrystalli commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1971191051
If it's of any use/related, this message just started showing up in some of our R CMD CHECK github actions of an R package that has arrow as a dependency. In tests of a function that uses arrow to read csv files (without loading the package) we now get a new message:
```
Message
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.
```
**only on windows**, causing comparison against our previous snapshots to fail because of the message: https://github.com/Infectious-Disease-Modeling-Hubs/hubValidations/actions/runs/8083994354/job/22088275404?pr=74
The last time the action was run and passed was Feb 13.
Let me know if you think this should be moved into a separate issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "aquasync (via GitHub)" <gi...@apache.org>.
aquasync commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1977780203
Ah ok, interesting, thanks for investigating. I did try a conda install but that was just for python & python packages, not R.
I'm just using a modified wheel in the interim with the manifest attached to all the dlls, but if the solution of renaming the R dll is already being used in the conda context it makes sense to just use that for default builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1945220383
Hi @aquasync, thanks for the report and your work getting to a cause and fix. I can reproduce what you saw in your original report on my Windows ARM machine so I can offer that as a start here. I'll work on getting some more eyes on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "Bisaloo (via GitHub)" <gi...@apache.org>.
Bisaloo commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-6060898213
Is this the same issue as https://github.com/apache/arrow/issues/20282?
We're getting hit by this as well in a workflow that uses both python & R.
https://github.com/rstudio/reticulate/issues/1541 seemed to propose a way forward. Is there anything blocking or anything we can do to help getting this implemented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "aquasync (via GitHub)" <gi...@apache.org>.
aquasync commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1942961404
After some more digging, I think that the cause is indeed the two "arrow.dll" files. Loading the python package first works because the R "arrow.dll" is loaded directly, whereas the pyarrow version is loaded indirectly by being dependencies of the .pyd files.
I tried a hacky solution of renaming the python one to "xrrow.dll", and running `sed -i.bak 's/arrow.dll/xrrow.dll/g'` on the .pyd files in the directory, and it now "works" when loading the R package first:
```
> library(arrow)
> library(reticulate)
...
> pa = import('pyarrow')
> pa
Module(pyarrow)
```
Perhaps the easiest solution then is for the R library to use a different name for its dll, though I'm not sure if it has to match the package name or if that is just a convention? Alternatively perhaps dll manifests can be used on the python side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "aquasync (via GitHub)" <gi...@apache.org>.
aquasync commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1942982807
I was able to get a manifest solution to work, following https://devblogs.microsoft.com/oldnewthing/20171011-00/?p=97195.
First I created "force_local_arrow.manifest" with the following:
```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<file name="arrow.dll" />
</assembly>
```
Then I patched each of "arrow_flight.dll", "arrow_python.dll", "lib.cp312-win_amd64.pyd", and "_hdfsio.cp312-win_amd64.pyd" (probably incomplete, just enough to load the package) as follows:
```
> mt.exe -manifest force_local_arrow.manifest -outputresource:lib.cp312-win_amd64.pyd;#2
```
Perhaps this approach is better than just renaming the R dll; by amending the build process for the windows python wheel to include the necessary flags to embed the manifest and avoid the problem altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1975426266
Hey @aquasync, I did a little bit more research on this today and have a few notes:
I found [this older Jira ticket](https://issues.apache.org/jira/browse/ARROW-8281) that seemed related. That led me to try installing R arrow from conda instead of CRAN and I was able to get things to work:
```sh
$ conda create -n reticulate-test -c conda-forge python==3.12 r-reticulate r-arrow pyarrow
$ conda activate reticulate-test
$ R
> library(arrow)
> library(reticulate)
> import("pyarrow")
Module(pyarrow)
```
So that's possibly a workaround in case you need one right now.
As for next steps, it seems like we can modify either the Python build or the R build here and my preference would be to modify the R build as it would be relatively low impact and we know it already works because of our conda package. And we'd only have to modify it for Windows. @assignUser do you have any immediate reaction or insights into this idea?
Before I figured the above out, I filed [an issue over on the reticulate repo](https://github.com/rstudio/reticulate/issues/1541) to see if they have any ideas so I'll be curious to see if they have any thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R][Python] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1975372105
Hi @annakrystalli, I don't think that's related but feel free to file a new issue and tag me. It doesn't look like anything related to that code has changed recently so my guess is that something in your environment may have changed. tzdb is a `Suggests` and getting this warning is expected on Windows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to load pyarrow package through reticulate in R on windows [arrow]
Posted by "aquasync (via GitHub)" <gi...@apache.org>.
aquasync commented on issue #40073:
URL: https://github.com/apache/arrow/issues/40073#issuecomment-1942918452
I've attempted to diagnose it by loading the library directly as follows:
```
> library(arrow)
...
> reticulate::repl_python()
Python 3.12.2 (C:/Program Files/Python312/python.exe)
Reticulate 1.35.0 REPL -- A Python interpreter in R.
Enter 'exit' or 'quit' to exit the REPL and return to R.
>>> import ctypes as c
>>> dll = c.WinDLL(r'C:\Program Files\Python312\Lib\site-packages\pyarrow\lib.cp312-win_amd64.pyd')
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\PROGRA~1\PYTHON~2\Lib\ctypes\__init__.py", line 379, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 127] The specified procedure could not be found
```
That produces a pop up with the mangled C++ symbol name that caused the issues, in this case eg: "The procedure entry point ?type@StringBuilder@arrow@@UEBA?AV?$shared_ptr@VDataType@arrow@@@std@@XZ could not be located in the dynamic link library C:\Program Files\Python312\Lib\site-packages\pyarrow\lib.cp312-win_amd64.pyd".
If I repeat the above without first loading the R arrow package, it will load the python libs just fine. Is this problem caused by both the R and python packages having different "arrow.dll" files that they're attempting to load into the same address space?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org