You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "dpengpeng (via GitHub)" <gi...@apache.org> on 2025/04/08 09:29:35 UTC
[I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
dpengpeng opened a new issue, #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258
### Backend
VL (Velox)
### Bug description
I compiled a Gluten 1.3.0 release version using the Centos7 image. I enabled the HDFS option and the compilation command is as follows:
`bash ./dev/buildbundle-veloxbe.sh --enable_hdfs=ON --enable_vcpkg=ON --spark_version=3.4 --build_arrow=OFF`
I created an Iceberg table on HDFS and now use Spark to query the table data. When I run the Gluten program in the K8S environment, I also use the Centos 7 image, and the following exception information is displayed. Why does it show that libhdfs.so cannot be found here? I have used the --enable_hdfs=ON switch.
```
E20250408 16:29:51.768519 88 HdfsFileSystem.cpp:35] ConnectLibHdfs failed due to: IOError: Unable to load libhdfs: ./libhdfs.so: cannot open shared object file: No such file or directory
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007f41a4273850, pid=20, tid=0x00007f41ea17f700
JRE version: OpenJDK Runtime Environment (8.0_422-b05) (build 1.8.0_422-b05)
Java VM: OpenJDK 64-Bit Server VM (25.422-b05 mixed mode linux-amd64 compressed oops)
Problematic frame:
C [libvelox.so+0x2776850] facebook::velox::filesystems::arrow::io::internal::LibHdfsShim::NewBuilder()+0x0
Core dump written. Default location: /opt/spark/work-dir/core or core.20
An error report file with more information is saved as:
/opt/spark/work-dir/hs_err_pid20.log
If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.
```
At the end of 2024, I compiled a Gluten program of version 1.3.0-SNAPSHOT, which can access HDFS normally. Now the release version seems to have changed on HDFS. How can I access HDFS now?
### Spark version
Spark-3.4.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "dpengpeng (via GitHub)" <gi...@apache.org>.
dpengpeng commented on issue #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258#issuecomment-2788162478
> This means your libhdfs.so is not in "$HADOOP_HOME/lib/native". You can add the environment variable ARROW_LIBHDFS_DIR=libhdfs.so path prefix. For example, for cdh, --conf "spark.executorEnv.ARROW_LIBHDFS_DIR=/opt/cloudera/parcels/CDH/lib64".
Thank you very much. I tried ARROW_LIBHDFS_DIR and LD_LIBRARY_PATH, both are OK. We can choose either one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "dpengpeng (via GitHub)" <gi...@apache.org>.
dpengpeng commented on issue #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258#issuecomment-2786448823
> As the document says, you still need to configure libhdfs.so at runtime, otherwise it will fail to load.
Thank you for your reply. Previously, I could compile version 1.3.0-SNAPSHOT (2024) with the --enable_hdfs=ON switch without configuring the ${HADOOP_HOME}/lib/native/libhdfs.so path at runtime. Since when is it mandatory to use libhdfs.so now? Is libhdfs3.so still needed? Can both libhdfs.so and libhdfs3.so still be integrated into the fat jar of gluten?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "dpengpeng (via GitHub)" <gi...@apache.org>.
dpengpeng commented on issue #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258#issuecomment-2786290112
> Can you check out the [hdfs-support](https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md#hdfs-support) document to see if it can help your?
I've looked into the "hdfs-support" related content, but I didn't figure out where the problem is. I downloaded libhdfs3.so through vcpkg during compilation, and used static compilation. Gluten should include the HDFS-related .so files, but the operation still fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on issue #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258#issuecomment-2786272685
Can you check out the [hdfs-support](https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md#hdfs-support) document to see if it can help your?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258#issuecomment-2788076808
This means your libhdfs.so is not in "$HADOOP_HOME/lib/native". You can add the environment variable ARROW_LIBHDFS_DIR=libhdfs.so path prefix. For example, for cdh, --conf "spark.executorEnv.ARROW_LIBHDFS_DIR=/opt/cloudera/parcels/CDH/lib64".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 closed issue #9258: [VL] Gluten cannot find libhdfs when accessing HDFS
URL: https://github.com/apache/incubator-gluten/issues/9258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Gluten cannot find libhdfs when accessing HDFS [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on issue #9258:
URL: https://github.com/apache/incubator-gluten/issues/9258#issuecomment-2786384308
As the document says, you still need to configure libhdfs.so at runtime, otherwise it will fail to load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org