You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:46:57 UTC
[GitHub] [arrow] mpeleshenko opened a new issue #7843: HDFS client unable to load libjvm on Windows
mpeleshenko opened a new issue #7843:
URL: https://github.com/apache/arrow/issues/7843
I've been having trouble connecting to HDFS even with the 1.0.0 pyarrow build as I run into the below error when running:
```
pa.hdfs.connect(host="host", port=port, user="user", kerb_ticket="kerb_ticket")
```
```
File "C:\ProgramData\Continuum\Anaconda\envs\pyarrow-test\lib\site-packages\pyarrow\hdfs.py", line 210 in connect
extra_conf=extra_conf)
File "C:\ProgramData\Continuum\Anaconda\envs\pyarrow-test\lib\site-packages\pyarrow\hdfs.py", line 40, in __init__
self._connect(host, port, user, kerb_ticket, extra_conf)
File "pyarrow\io-hdfs.pxi", line 75, in pyarrow.lib.HadoopFileSystem._connect
File "pyarrow\error.pxi", line 99, in pyarrow.lib.check_status
OSError: Unable to load libjvm: The specified module could not be found.
```
I tried the workaround mentioned in [ARROW-5236](https://issues.apache.org/jira/browse/ARROW-5236?focusedCommentId=17106888&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17106888) and got it working by copying jvm.dll into %JAVA_HOME%\lib\server\libjvm.so. It seems the logic to find libjvm is following a Linux path for some reason.
Looking into the arrow internals, I came across this:
https://github.com/apache/arrow/blob/b0d623957db820de4f1ff0a5ebd3e888194a48f0/cpp/src/arrow/io/hdfs_internal.cc#L176-L180
This looks like the same issue observed in [ARROW-1003](https://issues.apache.org/jira/browse/ARROW-1003), except that one was for libhdfs. In my situation, libhdfs is found as expected as hdfs.dll, so Windows logic is definitely followed there.
https://github.com/apache/arrow/blob/b0d623957db820de4f1ff0a5ebd3e888194a48f0/cpp/src/arrow/io/hdfs_internal.cc#L144-L145
I suspect a similar fix is needed here to change `__WIN32` to `_WIN32`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm closed issue #7843: HDFS client unable to load libjvm on Windows
Posted by GitBox <gi...@apache.org>.
wesm closed issue #7843:
URL: https://github.com/apache/arrow/issues/7843
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mpeleshenko commented on issue #7843: HDFS client unable to load libjvm on Windows
Posted by GitBox <gi...@apache.org>.
mpeleshenko commented on issue #7843:
URL: https://github.com/apache/arrow/issues/7843#issuecomment-664668096
@wesm This is actually the same issue described in [ARROW-5236](https://issues.apache.org/jira/browse/ARROW-5236). Can we use that one or should I still create a new JIRA issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on issue #7843: HDFS client unable to load libjvm on Windows
Posted by GitBox <gi...@apache.org>.
wesm commented on issue #7843:
URL: https://github.com/apache/arrow/issues/7843#issuecomment-664667146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org