You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "KAlbert2333 (via GitHub)" <gi...@apache.org> on 2025/08/05 03:59:51 UTC

[I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

KAlbert2333 opened a new issue, #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   [Expected behavior] WARN GlutenFallbackReporter: Validation failed for plan: ColumnarToRow[QueryId=50], due to: Scheme of [List(hdfs://xx.xx.xx.xx:9000/tpcds/parquet/store_returns)] is not supported by registered file systems..
   
   ### Gluten version
   
   Gluten-1.3
   
   ### Spark version
   
   Spark-3.4.x
   
   ### Spark configurations
   
   ${SPARK_HOME}/bin/spark-sql \
     --master local[*] \
     --driver-memory 50g \
     --executor-memory 400g \
     --conf spark.driver.extraJavaOptions="-XX:+UseNUMA -XX:+UseG1GC -Dlog4j.logger.org.apache.hadoop=DEBUG" \
     --conf spark.executor.extraJavaOptions="-XX:+UseNUMA -XX:+UseG1GC -Dlog4j.logger.org.apache.hadoop=DEBUG" \
     --conf spark.sql.shuffle.partitions=200 \
     --conf spark.memory.offHeap.enabled=true \
     --conf spark.memory.offHeap.size=50G \
     --conf spark.driver.extraClassPath=${SPARK_DRIVER_EXTRA_CLASSPATH} \
     --conf spark.executor.extraClassPath=${SPARK_EXECUTOR_EXTRA_CLASSPATH} \
     --conf spark.plugins=org.apache.gluten.GlutenPlugin \
     --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
     --conf spark.executorEnv.LIBHDFS3_CONF="hdfs-client.xml" \
     --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
     --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
     --conf spark.sql.adaptive.enabled=false \
     -f /data/tpcds-1tb/tools/run_all.sql \
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "KAlbert2333 (via GitHub)" <gi...@apache.org>.
KAlbert2333 commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3191101866

   My SSD hard drive seems to have a problem, but after replacing it, the release version runs normally


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3181860221

   @luofeng1994 You need to enable hdfs in your compile command. Could you help try following command?
   
   `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON --enable_hdfs=ON`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3166284205

   @KAlbert2333 Could you please show me the compile command you used?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3163124600

   @KAlbert2333 it looks odd, as HDFS support should be there for a long time. @JkSelf do you have any insights on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "luofeng1994 (via GitHub)" <gi...@apache.org>.
luofeng1994 commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3178574597

   I meet the same problem.
   
   my compile command:
   
   `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON`
   
   when run sparksql with the compiled gluten jar, hdfs scan orc fallback.
   `25/08/12 16:54:10 WARN GlutenFallbackReporter: Validation failed for plan: Scan orc tpcds_sf10.customer[QueryId=10], due to:
    - Scheme of [List(hdfs://master:8020/xxx/xxx/xxx/tpcds_10/customer)] is not supported by registered file systems.`
   
   how to solve this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "KAlbert2333 (via GitHub)" <gi...@apache.org>.
KAlbert2333 commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3163912170

   > [@KAlbert2333](https://github.com/KAlbert2333) it looks odd, as HDFS support should be there for a long time. [@JkSelf](https://github.com/JkSelf) do you have any insights on this?
   
   After I compiled it into debug mode, it worked fine, but I don't understand


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "luofeng1994 (via GitHub)" <gi...@apache.org>.
luofeng1994 commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3182473015

   > [@luofeng1994](https://github.com/luofeng1994) You need to enable hdfs in your compile command. Could you help try following command?
   > 
   > `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON --enable_hdfs=ON`
   
   I tried this command, but is strange. 
   
   When I ran this command, the build process ended after packaging the gluten jar. I got the new gluten jar at package/target.
   `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON`
   
   But when I ran with the parameter --enable_hdfs=ON, the build process ended after build the libgluten.so libvelox.so, without any error. The gluten jar at package/target not updated.
   `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON --enable_hdfs=ON`
   
   the log ended like this
   `
   + build_gluten_cpp
   + echo 'Start to build Gluten CPP'
   Start to build Gluten CPP
   + cd /root/luof/incubator-gluten/dev/../cpp
   + rm -rf build
   + mkdir build
   + cd build
   + GLUTEN_CMAKE_OPTIONS='-DBUILD_VELOX_BACKEND=ON     -DCMAKE_BUILD_TYPE=Release     -DVELOX_HOME=/root/luof/incubator-gluten/dev/../ep/build-velox/build/velox_ep     -DBUILD_TESTS=OFF     -DBUILD_EXAMPLES=OFF     -DBUILD_BENCHMARKS=OFF     -DENABLE_JEMALLOC_STATS=OFF     -DENABLE_HBM=OFF     -DENABLE_QAT=OFF     -DENABLE_IAA=OFF     -DENABLE_GCS=OFF     -DENABLE_S3=OFF     -DENABLE_HDFS=ON     -DENABLE_ABFS=OFF     -DENABLE_GPU=OFF     -DENABLE_ENHANCED_FEATURES=OFF'
   + '[' Linux == Darwin ']'
   + cmake -DBUILD_VELOX_BACKEND=ON -DCMAKE_BUILD_TYPE=Release -DVELOX_HOME=/root/luof/incubator-gluten/dev/../ep/build-velox/build/velox_ep -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARKS=OFF -DENABLE_JEMALLOC_STATS=OFF -DENABLE_HBM=OFF -DENABLE_QAT=OFF -DENABLE_IAA=OFF -DENABLE_GCS=OFF -DENABLE_S3=OFF -DENABLE_HDFS=ON -DENABLE_ABFS=OFF -DENABLE_GPU=OFF -DENABLE_ENHANCED_FEATURES=OFF ..
   -- Building using CMake version: 3.28.3
   -- The C compiler identification is GNU 11.1.0
   -- The CXX compiler identification is GNU 11.1.0
   -- Detecting C compiler ABI info
   -- Detecting C compiler ABI info - done
   -- Check for working C compiler: /usr/bin/cc - skipped
   -- Detecting C compile features
   -- Detecting C compile features - done
   -- Detecting CXX compiler ABI info
   -- Detecting CXX compiler ABI info - done
   -- Check for working CXX compiler: /usr/bin/c++ - skipped
   -- Detecting CXX compile features
   -- Detecting CXX compile features - done
   -- Add definition NDEBUG
   -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
   -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
   -- Check if compiler accepts -pthread
   -- Check if compiler accepts -pthread - yes
   -- Found Threads: TRUE
   -- Found JNI: /usr/jdk64/jdk1.8.0_461/include  found components: AWT JVM
   -- Found glog: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libglog.a
   -- Found PkgConfig: /usr/bin/pkg-config (found version "1.7.3")
   Core module final CMAKE_CXX_FLAGS= -Wno-class-memaccess -Wno-error=unused-but-set-variable       -Wall        -Wno-sign-compare        -Wno-comment        -Werror        -Wno-error=parentheses        -Wno-error=unused-function        -Wno-error=unused-variable        -Wno-strict-aliasing        -Wno-ignored-qualifiers        -Wno-deprecated-declarations        -Wno-attributes -march=armv8-a+crc+crypto
   -- Found Boost: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include (found version "1.84.0")
   -- Set Gluten Proto Directory in /root/luof/incubator-gluten/gluten-core/src/main/resources/org/apache/gluten/proto
   -- Set Substrait Proto Directory in /root/luof/incubator-gluten/gluten-substrait/src/main/resources/substrait/proto
   -- Found Arrow library: /usr/local/lib64/libarrow.a
   -- Found Arrow library: /usr/local/lib64/libarrow_bundled_dependencies.a
   -- Found Protobuf: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libprotobuf.a;-pthread (found version "3.21.12")
   -- Found Protobuf: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libprotobuf.a
   Velox module final CMAKE_CXX_FLAGS= -Wno-class-memaccess -Wno-error=unused-but-set-variable       -Wall        -Wno-sign-compare        -Wno-comment        -Werror        -Wno-error=parentheses        -Wno-error=unused-function        -Wno-error=unused-variable        -Wno-strict-aliasing        -Wno-ignored-qualifiers        -Wno-deprecated-declarations        -Wno-attributes -march=armv8-a+crc+crypto
   -- Set Gluten Proto Directory in /root/luof/incubator-gluten/backends-velox/src/main/resources/org/apache/gluten/proto
   -- Found Protobuf: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libprotobuf.a
   -- Found libevent include directory: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include
   -- Found libevent component: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libevent_core.a
   -- Found libevent component: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libevent_extra.a
   -- Found libevent component: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libevent_pthreads.a
   -- Found libevent 2.2.0 in /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon
   -- Found ZLIB: optimized;/root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libz.a;debug;/root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/debug/lib/libz.a (found version "1.2.13")
   -- Found OpenSSL: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libcrypto.a (found version "3.0.8")
   -- Found Boost: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include (found version "1.84.0") found components: context filesystem program_options regex system thread chrono atomic
   -- Found folly: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon
   -- RE2 Library Can Be Found in re2::re2
   -- Found stemmer: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libstemmer.a
   -- Found the following ICU libraries:
   --   i18n (required): /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libicui18n.a
   --   uc (required): /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libicuuc.a
   --   data (required): /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/lib/libicudata.a
   -- Found ICU: /root/luof/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-neon/include (found version "72.1")
   -- Configuring done (1.4s)
   -- Generating done (0.2s)
   -- Build files have been written to: /root/luof/incubator-gluten/cpp/build
   + make -j 2
   [  1%] Running Gluten Velox PROTO compiler
   [  2%] Running Substrait PROTO compiler
   [  2%] Built target velox_jni_proto
   [  3%] Running Gluten PROTO compiler
   [  3%] Built target jni_proto
   [  4%] Building CXX object core/CMakeFiles/gluten.dir/proto/substrait/algebra.pb.cc.o
   [  5%] Building CXX object core/CMakeFiles/gluten.dir/proto/substrait/capabilities.pb.cc.o
   [  6%] Building CXX object core/CMakeFiles/gluten.dir/proto/substrait/extended_expression.pb.cc.o
   
   ……
   
   [ 97%] Building CXX object velox/CMakeFiles/velox.dir/utils/VeloxWholeStageDumper.cc.o
   [ 98%] Linking CXX shared library ../releases/libvelox.so
   [100%] Built target velox
   [root@slave2 incubator-gluten]#
   `
   
   How can I complete the gluten packaging after adding --enable_hdfs=ON ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3159933604

   @KAlbert2333 
   how did you compile the gluten binary? `--enable_hdfs=ON` should be kept to enable hdfs support 
   https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md#hdfs-support
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "KAlbert2333 (via GitHub)" <gi...@apache.org>.
KAlbert2333 commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3162179020

   > [@KAlbert2333](https://github.com/KAlbert2333) how did you compile the gluten binary? `--enable_hdfs=ON` should be kept to enable hdfs support https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md#hdfs-support
   
   I turned on the HDFS option, but it doesn't work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "luofeng1994 (via GitHub)" <gi...@apache.org>.
luofeng1994 commented on issue #10360:
URL: https://github.com/apache/incubator-gluten/issues/10360#issuecomment-3183171399

   > [@luofeng1994](https://github.com/luofeng1994) You need to enable hdfs in your compile command. Could you help try following command?
   > 
   > `./dev/builddeps-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF --enable_vcpkg=ON --enable_hdfs=ON`
   
   It works! After rebuild libvelox.so by adding  `--enable_hdfs=ON` and repackage the gluten jar, the problem solved. tks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] BatchScan operator fallback when enable hdfs file system [incubator-gluten]

Posted by "KAlbert2333 (via GitHub)" <gi...@apache.org>.
KAlbert2333 closed issue #10360: BatchScan operator fallback when enable hdfs file system
URL: https://github.com/apache/incubator-gluten/issues/10360


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org