You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/18 03:29:26 UTC
[GitHub] [arrow] h-vetinari opened a new issue, #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
h-vetinari opened a new issue, #15019:
URL: https://github.com/apache/arrow/issues/15019
### Describe the bug, including details regarding any error messages, version, and platform.
#14832 switched on the test suite within the conda-builds, which itself synced with https://github.com/conda-forge/arrow-cpp-feedstock/pull/875
In the process of doing so, I removed
https://github.com/apache/arrow/blob/2c768a1767c0d7b33644910e0e96a949bc6f6f42/dev/tasks/conda-recipes/arrow-cpp/meta.yaml#L351-L353
because the tests always showed: `SKIPPED [24] test_hdfs.py:48: No libhdfs available on system`
Even with a patch as follows:
```
diff --git a/cpp/src/arrow/io/hdfs_internal.cc b/cpp/src/arrow/io/hdfs_internal.cc
index 4592392b8..9f8f70389 100644
--- a/cpp/src/arrow/io/hdfs_internal.cc
+++ b/cpp/src/arrow/io/hdfs_internal.cc
@@ -144,9 +144,9 @@ Result<std::vector<PlatformFilename>> get_potential_libhdfs_paths() {
#ifdef _WIN32
file_name = "hdfs.dll";
#elif __APPLE__
- file_name = "libhdfs.dylib";
+ file_name = "libhdfs3.dylib";
#else
- file_name = "libhdfs.so";
+ file_name = "libhdfs3.so";
#endif
// Common paths
@@ -155,6 +155,8 @@ Result<std::vector<PlatformFilename>> get_potential_libhdfs_paths() {
// Path from environment variable
AppendEnvVarFilename("HADOOP_HOME", "lib/native", &search_paths);
AppendEnvVarFilename("ARROW_LIBHDFS_DIR", &search_paths);
+ AppendEnvVarFilename("CONDA_PREFIX", "lib", &search_paths);
+ AppendEnvVarFilename("LIBRARY_LIB", &search_paths);
// All paths with file name
for (const auto& path : search_paths) {
```
this remained the case. I thought this might be necessary because in the conda-forge world, the name of the libhdfs binary [contains](https://github.com/regro/libcfgraph/blob/master/artifacts/libhdfs3/conda-forge/linux-64/libhdfs3-2.3-hc2c6f8d_1022.json#L230) the "3", but there seem to be other issues at play here as well.
To fix this issue, the line quoted above should be uncommented, and the conda tests should not show `SKIPPED [24] test_hdfs.py:48: No libhdfs available on system` anymore (at least on unix).
### Component(s)
C++, Continuous Integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #15019:
URL: https://github.com/apache/arrow/issues/15019#issuecomment-1360992352
The Arrow HDFS implementation is based on the JNI libhdfs, so it is expected that it doesn't work with libhdfs3. And since libhdfs doesn't seem to be packages by conda-forge, I don't think there is a way to run the hdfs tests in the conda-forge build purely based on conda packages (our own tests install the JNI library manually on top of the conda env, see eg ci/docker/conda-python-hdfs.dockerfile)
We have had integration with libhdfs3 as well in the past (and you could switch between both drivers), but this was removed almost 3 years ago (https://github.com/apache/arrow/pull/6432), because the libhdfs3 project is unmaintained. Also the dask filesystem wrapper using libhdfs3 is archived (https://github.com/dask/hdfs3).
Unless the libhdfs3 project would be revived, I don't think we should currently consider adding support for it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed issue #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
Posted by GitBox <gi...@apache.org>.
kou closed issue #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
URL: https://github.com/apache/arrow/issues/15019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
Posted by GitBox <gi...@apache.org>.
kou commented on issue #15019:
URL: https://github.com/apache/arrow/issues/15019#issuecomment-1362034989
> but this was removed almost 3 years ago (https://github.com/apache/arrow/pull/6432)
Oh, the author of the pull request is me... I didn't remember it...
I close this because we don't support `libhdfs3`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
Posted by GitBox <gi...@apache.org>.
kou commented on issue #15019:
URL: https://github.com/apache/arrow/issues/15019#issuecomment-1357014875
It seems that `libhdfs.so` and `libhdfs3.so` are different project.
`libhdfs.so` is a Java (JNI) library but `libhdfs3.so` is not.
https://github.com/martindurant/libhdfs3-downstream/tree/master/libhdfs3
I think that we need to implement `libhdfs3.so` based HDFS backend instead of reusing the current `libhdfs.so` based implementation. (I think that this is a feature request not a bug.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #15019: BUG: `libhdfs3` not getting picked up correctly when using conda
Posted by GitBox <gi...@apache.org>.
h-vetinari commented on issue #15019:
URL: https://github.com/apache/arrow/issues/15019#issuecomment-1361019379
Cool, thanks for the input. I wasn't aware of the split between `libhdfs` / `libhdfs3`; in this case I think it might be worthwhile medium term to package `libhdfs` in conda-forge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org