You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhouyuan (via GitHub)" <gi...@apache.org> on 2024/03/25 05:43:24 UTC
[I] [VL] Supporting JVM based libhdfs connector [incubator-gluten]
zhouyuan opened a new issue, #5103:
URL: https://github.com/apache/incubator-gluten/issues/5103
### Description
Velox hdfs connector is using libhdfs3 which is a C++ based hdfs client. The libhdfs3 is from HAWQ project and is not maintained anymore.
In Gluten we used the CK libhdfs3 fork, and added the delegation token support. (https://github.com/oap-project/libhdfs3)
However there are still some feature gaps like encrypted rpc(https://github.com/ClickHouse/ClickHouse/issues/22236), federated cluster
In practice the HDFS team and Spark team is usually separated so it maybe difficult to introduce a new HDFS client.
Gluten/Velox may also need to support JVM based libhdfs (hadoop default) as the second option, so there's no feature gap in HDFS connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Supporting JVM based libhdfs connector [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma closed issue #5103: [VL] Supporting JVM based libhdfs connector
URL: https://github.com/apache/incubator-gluten/issues/5103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org