You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Chris Dunderdale (Jira)" <ji...@apache.org> on 2022/03/11 16:27:00 UTC
[jira] [Commented] (ARROW-13329) [R][Python] DLL ImportError loading pyarrow with reticulate
[ https://issues.apache.org/jira/browse/ARROW-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504994#comment-17504994 ]
Chris Dunderdale commented on ARROW-13329:
------------------------------------------
Just ran into the same problem today, unfortunately. I haven't got a solution for this, but I can replicate a similar scenario using the following steps:
# Start with fresh windows (newly spun up VM on Azure) - Windows 10 20h2-pro-g2
# Download and install R (4.13) from CRAN
# Download and install RTools40 (asked for by reticulate/arrow packages when installing)
# As per RTools website, set the .REnviron for the make.exe path
# Download and install RStudio 2022.02.0 Build 443 from the RStudio website
# In R
{code:java}
install.packages(reticulate)
### Details
package ‘rprojroot’ successfully unpacked and MD5 sums checked
package ‘Rcpp’ successfully unpacked and MD5 sums checked
package ‘RcppTOML’ successfully unpacked and MD5 sums checked
package ‘here’ successfully unpacked and MD5 sums checked
package ‘jsonlite’ successfully unpacked and MD5 sums checked
package ‘png’ successfully unpacked and MD5 sums checked
package ‘rappdirs’ successfully unpacked and MD5 sums checked
package ‘withr’ successfully unpacked and MD5 sums checked
package ‘reticulate’ successfully unpacked and MD5 sums checked
{code}
#
{code:java}
install.packages(arrow)
### Details
package ‘bit’ successfully unpacked and MD5 sums checked
package ‘magrittr’ successfully unpacked and MD5 sums checked
package ‘ellipsis’ successfully unpacked and MD5 sums checked
package ‘glue’ successfully unpacked and MD5 sums checked
package ‘assertthat’ successfully unpacked and MD5 sums checked
package ‘bit64’ successfully unpacked and MD5 sums checked
package ‘purrr’ successfully unpacked and MD5 sums checked
package ‘R6’ successfully unpacked and MD5 sums checked
package ‘rlang’ successfully unpacked and MD5 sums checked
package ‘tidyselect’ successfully unpacked and MD5 sums checked
package ‘vctrs’ successfully unpacked and MD5 sums checked
package ‘cpp11’ successfully unpacked and MD5 sums checked
package ‘arrow’ successfully unpacked and MD5 sums checked{code}
#
{code:java}
library(reticulate)
library(arrow){code}
#
{code:java}
install_miniconda() #respond y to install
### Details
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done## Package Plan ## environment location: C:\Users\<user>~1\AppData\Local\R-MINI~1 added / updated specs:
- conda
The following packages will be downloaded: package | build
---------------------------|-----------------
ca-certificates-2022.2.1 | haa95532_0 123 KB
cffi-1.15.0 | py39h2bbff1b_1 224 KB
pyopenssl-22.0.0 | pyhd3eb1b0_0 50 KB
six-1.16.0 | pyhd3eb1b0_1 18 KB
sqlite-3.38.0 | h2bbff1b_0 807 KB
urllib3-1.26.8 | pyhd3eb1b0_0 106 KB
------------------------------------------------------------
Total: 1.3 MBThe following packages will be REMOVED: conda-content-trust-0.1.1-pyhd3eb1b0_0The following packages will be UPDATED: ca-certificates 2021.10.26-haa95532_2 --> 2022.2.1-haa95532_0
cffi 1.15.0-py39h2bbff1b_0 --> 1.15.0-py39h2bbff1b_1
pyopenssl 21.0.0-pyhd3eb1b0_1 --> 22.0.0-pyhd3eb1b0_0
six 1.16.0-pyhd3eb1b0_0 --> 1.16.0-pyhd3eb1b0_1
sqlite 3.37.0-h2bbff1b_0 --> 3.38.0-h2bbff1b_0
urllib3 1.26.7-pyhd3eb1b0_0 --> 1.26.8-pyhd3eb1b0_0Downloading and Extracting Packages
pyopenssl-22.0.0 | 50 KB | ########## | 100%
urllib3-1.26.8 | 106 KB | ########## | 100%
sqlite-3.38.0 | 807 KB | ########## | 100%
cffi-1.15.0 | 224 KB | ########## | 100%
ca-certificates-2022 | 123 KB | ########## | 100%
six-1.16.0 | 18 KB | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done## Package Plan ## environment location: C:\Users\<user>~1\AppData\Local\R-MINI~1\envs\r-reticulate added / updated specs:
- numpy
- python=3.8
The following packages will be downloaded: package | build
---------------------------|-----------------
ca-certificates-2021.10.8 | h5b45459_0 176 KB conda-forge
intel-openmp-2022.0.0 | h57928b3_3663 3.6 MB conda-forge
libblas-3.9.0 | 13_win64_mkl 5.3 MB conda-forge
libcblas-3.9.0 | 13_win64_mkl 5.3 MB conda-forge
liblapack-3.9.0 | 13_win64_mkl 5.3 MB conda-forge
mkl-2022.0.0 | h0e2418a_796 181.9 MB conda-forge
numpy-1.22.3 | py38hcf66579_0 6.1 MB conda-forge
openssl-3.0.0 | h8ffe710_2 10.0 MB conda-forge
pip-22.0.4 | pyhd8ed1ab_0 1.5 MB conda-forge
python-3.8.12 |h900ac77_2_cpython 19.3 MB conda-forge
python_abi-3.8 | 2_cp38 4 KB conda-forge
setuptools-60.9.3 | py38haa244fe_0 1.1 MB conda-forge
sqlite-3.37.0 | h8ffe710_0 1.2 MB conda-forge
tbb-2021.5.0 | h2d74725_0 148 KB conda-forge
ucrt-10.0.20348.0 | h57928b3_0 1.2 MB conda-forge
vc-14.2 | hb210afc_6 13 KB conda-forge
vs2015_runtime-14.29.30037 | h902a5da_6 1.3 MB conda-forge
wheel-0.37.1 | pyhd8ed1ab_0 31 KB conda-forge
------------------------------------------------------------
Total: 243.3 MBThe following NEW packages will be INSTALLED: ca-certificates conda-forge/win-64::ca-certificates-2021.10.8-h5b45459_0
intel-openmp conda-forge/win-64::intel-openmp-2022.0.0-h57928b3_3663
libblas conda-forge/win-64::libblas-3.9.0-13_win64_mkl
libcblas conda-forge/win-64::libcblas-3.9.0-13_win64_mkl
liblapack conda-forge/win-64::liblapack-3.9.0-13_win64_mkl
mkl conda-forge/win-64::mkl-2022.0.0-h0e2418a_796
numpy conda-forge/win-64::numpy-1.22.3-py38hcf66579_0
openssl conda-forge/win-64::openssl-3.0.0-h8ffe710_2
pip conda-forge/noarch::pip-22.0.4-pyhd8ed1ab_0
python conda-forge/win-64::python-3.8.12-h900ac77_2_cpython
python_abi conda-forge/win-64::python_abi-3.8-2_cp38
setuptools conda-forge/win-64::setuptools-60.9.3-py38haa244fe_0
sqlite conda-forge/win-64::sqlite-3.37.0-h8ffe710_0
tbb conda-forge/win-64::tbb-2021.5.0-h2d74725_0
ucrt conda-forge/win-64::ucrt-10.0.20348.0-h57928b3_0
vc conda-forge/win-64::vc-14.2-hb210afc_6
vs2015_runtime conda-forge/win-64::vs2015_runtime-14.29.30037-h902a5da_6
wheel conda-forge/noarch::wheel-0.37.1-pyhd8ed1ab_0
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
* Miniconda has been successfully installed at "C:/Users/<user>/AppData/Local/r-miniconda".{code}
#
{code:java}
py_config()
### Details
python: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate/python38.dll
pythonhome: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate
version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version: 1.22.3{code}
#
{code:java}
#Try connect to client even though arrow is not installed on env
client <- flight_connect(port = 5005)
#expected error
Error in py_module_import(module, convert = convert) :
ModuleNotFoundError: No module named 'pyarrow'{code}
#
{code:java}
install_pyarrow()
### Details
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done## Package Plan ## environment location: C:\Users\DynamoAdmin\AppData\Local\r-miniconda\envs\r-reticulate added / updated specs:
- pyarrow
The following packages will be downloaded: package | build
---------------------------|-----------------
arrow-cpp-3.0.0 |py38haeefe89_1_cpu 13.7 MB conda-forge
aws-c-common-0.4.57 | ha925a31_1 150 KB conda-forge
aws-c-event-stream-0.1.6 | h7915e17_3 26 KB conda-forge
aws-checksums-0.1.9 | hb01e4cc_0 51 KB conda-forge
aws-sdk-cpp-1.8.130 | hd77b12b_0 2.4 MB
brotli-1.0.9 | h8ffe710_6 18 KB conda-forge
brotli-bin-1.0.9 | h8ffe710_6 21 KB conda-forge
bzip2-1.0.8 | h8ffe710_4 149 KB conda-forge
c-ares-1.18.1 | h8ffe710_0 114 KB conda-forge
gflags-2.2.2 | ha925a31_1004 80 KB conda-forge
glog-0.4.0 | h0174b99_3 83 KB conda-forge
grpc-cpp-1.35.0 | hc60d5dd_4 6.8 MB
krb5-1.19.2 | hc8ab02b_4 847 KB conda-forge
libbrotlicommon-1.0.9 | h8ffe710_6 67 KB conda-forge
libbrotlidec-1.0.9 | h8ffe710_6 33 KB conda-forge
libbrotlienc-1.0.9 | h8ffe710_6 721 KB conda-forge
libcurl-7.82.0 | h789b8ee_0 301 KB conda-forge
libevent-2.1.10 | h91dae50_4 422 KB conda-forge
libprotobuf-3.14.0 | h7755175_0 2.3 MB conda-forge
libssh2-1.10.0 | h9a1e1f7_2 227 KB conda-forge
libthrift-0.13.0 | he1d8c1a_6 1.1 MB
libutf8proc-2.7.0 | hcb41399_0 101 KB conda-forge
libzlib-1.2.11 | h8ffe710_1013 63 KB conda-forge
lz4-c-1.9.3 | h8ffe710_1 135 KB conda-forge
parquet-cpp-1.5.1 | 2 3 KB conda-forge
pyarrow-3.0.0 |py38ha37a76c_1_cpu 2.1 MB conda-forge
re2-2020.11.01 | h0e60522_0 468 KB conda-forge
snappy-1.1.8 | ha925a31_3 50 KB conda-forge
xz-5.2.5 | h62dcd97_1 211 KB conda-forge
zlib-1.2.11 | h8ffe710_1013 106 KB conda-forge
zstd-1.4.9 | h6255e5f_0 915 KB conda-forge
------------------------------------------------------------
Total: 33.6 MBThe following NEW packages will be INSTALLED: arrow-cpp conda-forge/win-64::arrow-cpp-3.0.0-py38haeefe89_1_cpu
aws-c-common conda-forge/win-64::aws-c-common-0.4.57-ha925a31_1
aws-c-event-stream conda-forge/win-64::aws-c-event-stream-0.1.6-h7915e17_3
aws-checksums conda-forge/win-64::aws-checksums-0.1.9-hb01e4cc_0
aws-sdk-cpp pkgs/main/win-64::aws-sdk-cpp-1.8.130-hd77b12b_0
brotli conda-forge/win-64::brotli-1.0.9-h8ffe710_6
brotli-bin conda-forge/win-64::brotli-bin-1.0.9-h8ffe710_6
bzip2 conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
c-ares conda-forge/win-64::c-ares-1.18.1-h8ffe710_0
gflags conda-forge/win-64::gflags-2.2.2-ha925a31_1004
glog conda-forge/win-64::glog-0.4.0-h0174b99_3
grpc-cpp pkgs/main/win-64::grpc-cpp-1.35.0-hc60d5dd_4
krb5 conda-forge/win-64::krb5-1.19.2-hc8ab02b_4
libbrotlicommon conda-forge/win-64::libbrotlicommon-1.0.9-h8ffe710_6
libbrotlidec conda-forge/win-64::libbrotlidec-1.0.9-h8ffe710_6
libbrotlienc conda-forge/win-64::libbrotlienc-1.0.9-h8ffe710_6
libcurl conda-forge/win-64::libcurl-7.82.0-h789b8ee_0
libevent conda-forge/win-64::libevent-2.1.10-h91dae50_4
libprotobuf conda-forge/win-64::libprotobuf-3.14.0-h7755175_0
libssh2 conda-forge/win-64::libssh2-1.10.0-h9a1e1f7_2
libthrift pkgs/main/win-64::libthrift-0.13.0-he1d8c1a_6
libutf8proc conda-forge/win-64::libutf8proc-2.7.0-hcb41399_0
libzlib conda-forge/win-64::libzlib-1.2.11-h8ffe710_1013
lz4-c conda-forge/win-64::lz4-c-1.9.3-h8ffe710_1
parquet-cpp conda-forge/noarch::parquet-cpp-1.5.1-2
pyarrow conda-forge/win-64::pyarrow-3.0.0-py38ha37a76c_1_cpu
re2 conda-forge/win-64::re2-2020.11.01-h0e60522_0
snappy conda-forge/win-64::snappy-1.1.8-ha925a31_3
xz conda-forge/win-64::xz-5.2.5-h62dcd97_1
zlib conda-forge/win-64::zlib-1.2.11-h8ffe710_1013
zstd conda-forge/win-64::zstd-1.4.9-h6255e5f_0Downloading and Extracting Packages
zlib-1.2.11 | 106 KB | ########## | 100%
c-ares-1.18.1 | 114 KB | ########## | 100%
libbrotlienc-1.0.9 | 721 KB | ########## | 100%
brotli-1.0.9 | 18 KB | ########## | 100%
krb5-1.19.2 | 847 KB | ########## | 100%
gflags-2.2.2 | 80 KB | ########## | 100%
aws-c-common-0.4.57 | 150 KB | ########## | 100%
libcurl-7.82.0 | 301 KB | ########## | 100%
grpc-cpp-1.35.0 | 6.8 MB | ########## | 100%
zstd-1.4.9 | 915 KB | ########## | 100%
aws-sdk-cpp-1.8.130 | 2.4 MB | ########## | 100%
re2-2020.11.01 | 468 KB | ########## | 100%
arrow-cpp-3.0.0 | 13.7 MB | ########## | 100%
libthrift-0.13.0 | 1.1 MB | ########## | 100%
xz-5.2.5 | 211 KB | ########## | 100%
brotli-bin-1.0.9 | 21 KB | ########## | 100%
bzip2-1.0.8 | 149 KB | ########## | 100%
libbrotlidec-1.0.9 | 33 KB | ########## | 100%
libevent-2.1.10 | 422 KB | ########## | 100%
libbrotlicommon-1.0. | 67 KB | ########## | 100%
aws-checksums-0.1.9 | 51 KB | ########## | 100%
aws-c-event-stream-0 | 26 KB | ########## | 100%
pyarrow-3.0.0 | 2.1 MB | ########## | 100%
lz4-c-1.9.3 | 135 KB | ########## | 100%
libutf8proc-2.7.0 | 101 KB | ########## | 100%
parquet-cpp-1.5.1 | 3 KB | ########## | 100%
libprotobuf-3.14.0 | 2.3 MB | ########## | 100%
libssh2-1.10.0 | 227 KB | ########## | 100%
libzlib-1.2.11 | 63 KB | ########## | 100%
snappy-1.1.8 | 50 KB | ########## | 100%
glog-0.4.0 | 83 KB | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done{code}
#
{code:java}
#just to confirm that nothing changed
py_config()
### Details
python: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate/python38.dll
pythonhome: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate
version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/Users/<user>Admin/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version: 1.22.3{code}
#
{code:java}
client <- flight_connect(port = 5005)
#Error in py_module_import(module, convert = convert) :
ImportError: DLL load failed while importing lib: The specified procedure could not be found.{code}
#
{code:java}
#try get any stack trace
traceback()
### Details
4: stop(structure(list(message = "ImportError: DLL load failed while importing lib: The specified procedure could not be found.",
call = py_module_import(module, convert = convert), cppstack = NULL), class = c("Rcpp::exception",
"C++Error", "error", "condition")))
3: py_module_import(module, convert = convert)
2: reticulate::import("pyarrow")
1: flight_connect(port = 5005){code}
I've tried this several times over the past few days with various configurations - all of which produce exactly the same issue
* With Anaconda instead of MiniConda that's installed in the above replication
* Without conda using a virtual env
* Making sure that my "base" interpreter doesn't have pyarrow installed instead of installing to my env
I can't seem to get this working by hook or by crook unfortunately :( would be great to know if someone has specific versions which they know are working.
> [R][Python] DLL ImportError loading pyarrow with reticulate
> -----------------------------------------------------------
>
> Key: ARROW-13329
> URL: https://issues.apache.org/jira/browse/ARROW-13329
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python, R
> Affects Versions: 4.0.1
> Reporter: Christian
> Priority: Major
>
> Hi,
>
> I'm trying to use arrow flight through reticulate but am getting the following error when calling any function from the library.
> Error in py_module_import(module, convert = convert) :
> ImportError: DLL load failed while importing lib: The specified procedure could not be found.
>
> I set up a virtual environment as such:
> virtualenv_create(
> envname ='...',
> python='C:/Users/.../AppData/Local/Programs/Python/Python39/python.exe',
> packages=c('pandas','xlsxwriter','pyarrow')
> )
> I can't use the arrow::install_pyarrow() command because I don't have conda installed.
>
> Any help would be appreciated.
>
> Regards
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)