You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2025/08/25 02:24:16 UTC
[I] Unable to connect to HDFS: , got error: GetLastExceptionRootCause return null. [incubator-gluten]
beliefer opened a new issue, #10519:
URL: https://github.com/apache/incubator-gluten/issues/10519
### Backend
VL (Velox)
### Bug description
I build the `libhdfs3.so` with `dev/build_libhdfs3.sh` at branch 1.4.0.
I put it into the `${HADOOP_HOME}/lib/native/` and then remove `libhdfs.so`.
At finally, I renamed `libhdfs3.so` to `libhdfs.so`.
I found the error show below when I was submitting Spark SQL.
```
Spark master: yarn, Application Id: application_1731493565474_0482
Time taken: 1.675 seconds
org.apache.spark.SparkException: Job aborted due to stage failure:
Aborting TaskSet 3.0 because task 25 (partition 25)
cannot run anywhere due to node and executor excludeOnFailure.
Most recent failure:
Lost task 25.0 in stage 3.0 (TID 2030) (host1 executor 3): org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Error during calling Java code from native code: org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Unable to connect to HDFS: cluster, got error: GetLastExceptionRootCause return null.
Retriable: False
Expression: hdfsClient_ != nullptr
Context: Split [Hive: hdfs://cluster/user/hadoop/performance-datasets/tpcds/sf1000-parquet/useDecimal=false,useDate=false,filterNull=false/customer/part-00000-ffaa28bd-4482-4724-b6fc-ebde6b750cef-c000.snappy.parquet 104857600 - 4194304] Task Gluten_Stage_3_TID_2030_VTID_3
Additional Context: Operator: TableScan[0] 0
Function: Impl
File: /home/hadoop/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp
Line: 51
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems14HdfsFileSystem4ImplC1EPKNS0_6config10ConfigBaseERKNS1_19HdfsServiceEndpointE
# 4 _ZN8facebook5velox11filesystems14HdfsFileSystemC2ERKSt10shared_ptrIKNS0_6config10ConfigBaseEERKNS1_19HdfsServiceEndpointE
# 5 _ZN5folly15basic_once_flagINS_15SharedMutexImplILb0EvSt6atomicNS_24SharedMutexPolicyDefaultEEES2_E14call_once_slowIZZN8facebook5velox11filesystems23hdfsFileSystemGeneratorEvENKUlSt10shared_ptrIKNS8_6config10ConfigBaseEESt17basic_string_viewIcSt11char_traitsIcEEE_clESE_SI_EUlvE_JEEEvOT_DpOT0_
# 6 _ZZN8facebook5velox11filesystems23hdfsFileSystemGeneratorEvENKUlSt10shared_ptrIKNS0_6config10ConfigBaseEESt17basic_string_viewIcSt11char_traitsIcEEE_clES6_SA_.constprop.0
# 7 _ZNSt17_Function_handlerIFSt10shared_ptrIN8facebook5velox11filesystems10FileSystemEES0_IKNS2_6config10ConfigBaseEESt17basic_string_viewIcSt11char_traitsIcEEEZNS3_23hdfsFileSystemGeneratorEvEUlS9_SD_E_E9_M_invokeERKSt9_Any_dataOS9_OSD_
# 8 _ZN8facebook5velox11filesystems13getFileSystemESt17basic_string_viewIcSt11char_traitsIcEESt10shared_ptrIKNS0_6config10ConfigBaseEE
# 9 _ZN8facebook5velox19FileHandleGeneratorclERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKNS0_14FilePropertiesEPNS0_11filesystems4File7IoStatsE
# 10 _ZN8facebook5velox13CachedFactoryINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS0_10FileHandleENS0_19FileHandleGeneratorENS0_14FilePropertiesENS0_11filesystems4File7IoStatsENS0_15FileHandleSizerESt8equal_toIS7_ESt4hashIS7_EE8generateERKS7_PKSA_PSD_
# 11 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 12 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 13 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 14 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 15 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 16 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 17 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 18 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 19 _ZN6gluten24WholeStageResultIterator4nextEv
# 20 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 21 0x00007f32e1017ab4
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at org.apache.gluten.vectorized.ColumnarBatchInIterator.hasNext(ColumnarBatchInIterator.java:36)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:127)
at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:127)
at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:257)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:515)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1507)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:518)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Unable to connect to HDFS: , got error: GetLastExceptionRootCause return null. [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10519:
URL: https://github.com/apache/incubator-gluten/issues/10519#issuecomment-3219302171
it looks like the `dev/build_libhdfs3.sh` has some issue, likely due to some dependency issue. will verify locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Unable to connect to HDFS: , got error: GetLastExceptionRootCause return null. [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on issue #10519:
URL: https://github.com/apache/incubator-gluten/issues/10519#issuecomment-3218633013
ping @zhouyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Unable to connect to HDFS: , got error: GetLastExceptionRootCause return null. [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10519:
URL: https://github.com/apache/incubator-gluten/issues/10519#issuecomment-3219802861
I have built one here with minimal dependencies. Please check if this could fix the issue
[libhdfs3.tgz](https://github.com/user-attachments/files/21966937/libhdfs3.tgz)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Unable to connect to HDFS: , got error: GetLastExceptionRootCause return null. [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on issue #10519:
URL: https://github.com/apache/incubator-gluten/issues/10519#issuecomment-3264355914
I have build and tested successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Unable to connect to HDFS: , got error: GetLastExceptionRootCause return null. [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer closed issue #10519: [VL] Unable to connect to HDFS: <cluser>, got error: GetLastExceptionRootCause return null.
URL: https://github.com/apache/incubator-gluten/issues/10519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org