You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "A-little-bit-of-data (via GitHub)" <gi...@apache.org> on 2025/07/02 09:05:20 UTC
[I] Compiling 1.4.0 on X86 fails [incubator-gluten]
A-little-bit-of-data opened a new issue, #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102
### Problem description
When executing `./dev/buildbundle-veloxbe.sh --enable_s3=ON` to `mvn clean install -pl bom,maven/module-info-compiler-maven-plugin,vector -am -DskipTests -Drat.skip -Dmaven.gitcommitid.skip -Dcheckstyle.skip -Dassembly.skipAssembly`
, an error is reported at this step, and the error is as follows:
```
[INFO] Compiling 4 source files to /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/target/classes
[WARNING] Unable to autodetect 'javac' path, using 'javac' from the environment.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/ModuleInfoTestCompilerPlugin.java:[22,43] error: package org.apache.maven.plugins.annotations does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/ModuleInfoTestCompilerPlugin.java:[23,43] error: package org.apache.maven.plugins.annotations does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/ModuleInfoTestCompilerPlugin.java:[24,43] error: package org.apache.maven.plugins.annotations does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/ModuleInfoTestCompilerPlugin.java:[25,31] error: package org.apache.maven.project does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/BaseModuleInfoCompilerPlugin.java:[32,30] error: package org.apache.maven.plugin does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/BaseModuleInfoCompilerPlugin.java:[33,30] error: package org.apache.maven.plugin does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/BaseModuleInfoCompilerPlugin.java:[34,20] error: package org.glavo.mic does not exist
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/BaseModuleInfoCompilerPlugin.java:[39,59] error: cannot find symbol
symbol: class AbstractMojo
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/ModuleInfoTestCompilerPlugin.java:[30,1] error: cannot find symbol
symbol: class Mojo
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/ModuleInfoTestCompilerPlugin.java:[37,10] error: cannot find symbol
symbol: class MavenProject
location: class ModuleInfoTestCompilerPlugin
[ERROR] /usr/incubator-gluten-1.4.0/ep/_ep/arrow_ep/java/maven/module-info-compiler-maven-plugin/src/main/java/org/apache/arrow/maven/plugins/BaseModuleInfoCompilerPlugin.java:[47,31] error: cannot find symbol
symbol: class MojoExecutionException
location: class BaseModuleInfoCompilerPlugin
```
### System information
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
### CMake log
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3071852168
> Thank you very much for your reply. I currently see `https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md`. There are two ways to compile. Which one do you recommend for better compatibility?
If you build on centos7 and use on ubuntu20, you must use static link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3131279545
> > > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, If all CPU cores are used for compilation, ARM is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of `NUM_THREADS`.
> >
> >
> > [@kevinw66](https://github.com/kevinw66) Hello, I have a question. I have used the images of `vcpkg-centos-8, centos-8-jdk11, and 1.4-vcpkg-centos-8`, but there are these errors.
> > ```
> > [137/647] Building CXX object velox/buffer/CMakeFiles/velox.dir/__/vector/VectorStream.cpp.o
> > In file included from /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/xsimd.hpp:55,
> > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/base/SimdUtil.h:24,
> > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/RawVector.h:21,
> > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/Scratch.h:19,
> > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/ByteStream.h:18,
> > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/VectorStream.h:22,
> > from /incubator-gluten/ep/build-velox/build/velox_ep/velox/vector/VectorStream.cpp:16:
> > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<float, float> xsimd::sincos(float)’:
> > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:981:54: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> > 981 | inline std::pair<float, float> sincos(float val) noexcept
> > | ^~~~~~~~
> > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<double, double> xsimd::sincos(double)’:
> > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:988:57: note: parameter passing for argument of type ‘std::pair<double, double>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> > 988 | inline std::pair<double, double> sincos(double val) noexcept
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > What needs to be adjusted to compile properly? The following variables have been used,
> > ```
> > export VCPKG_FORCE_SYSTEM_BINARIES=1
> > export CPU_TARGET="aarch64"
> > export NUM_THREADS=4
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > My compiling machine is 8C
>
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) I think these are just warnings and based on my current testing, they do not affect our main functions, so I ignored them. These warnings also appears in our CI environment: https://github.com/apache/incubator-gluten/actions/runs/16491898666/job/46628360332#step:5:1799.
>
> <img alt="Image" width="1889" height="952" src="https://private-user-images.githubusercontent.com/31196226/470230088-1eb86399-1d6b-4c15-8a1e-6919f9d01032.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTM3Nzc3NTcsIm5iZiI6MTc1Mzc3NzQ1NywicGF0aCI6Ii8zMTE5NjIyNi80NzAyMzAwODgtMWViODYzOTktMWQ2Yi00YzE1LThhMWUtNjkxOWY5ZDAxMDMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzI5VDA4MjQxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhN2M4NGNkM2Y2NmIxOTJmYTQyMzI1NDNlYTFkY2IzOWM1Nzc5MTU1YzE0ZTc3MWU0ZDFlYzMzZjk4ZWZjNmMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.nQLGIcq6Q3Znw3OceowyqSorqSAFG8bQVCbppQTfaXU">
Okay, thanks for the tip. My ARM machine currently doesn't have enough memory to compile. I also have a question about the minimum memory required for this compilation. I can compile with less than 32GB on an x86 machine using the same image, but there are only 16GB machines for the ARM architecture. Can you give me a more accurate memory reference so I can apply for a machine? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3131294603
> > > > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, If all CPU cores are used for compilation, ARM is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of `NUM_THREADS`.
> > >
> > >
> > > [@kevinw66](https://github.com/kevinw66) Hello, I have a question. I have used the images of `vcpkg-centos-8, centos-8-jdk11, and 1.4-vcpkg-centos-8`, but there are these errors.
> > > ```
> > > [137/647] Building CXX object velox/buffer/CMakeFiles/velox.dir/__/vector/VectorStream.cpp.o
> > > In file included from /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/xsimd.hpp:55,
> > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/base/SimdUtil.h:24,
> > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/RawVector.h:21,
> > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/Scratch.h:19,
> > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/ByteStream.h:18,
> > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/VectorStream.h:22,
> > > from /incubator-gluten/ep/build-velox/build/velox_ep/velox/vector/VectorStream.cpp:16:
> > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<float, float> xsimd::sincos(float)’:
> > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:981:54: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> > > 981 | inline std::pair<float, float> sincos(float val) noexcept
> > > | ^~~~~~~~
> > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<double, double> xsimd::sincos(double)’:
> > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:988:57: note: parameter passing for argument of type ‘std::pair<double, double>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> > > 988 | inline std::pair<double, double> sincos(double val) noexcept
> > > ```
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > What needs to be adjusted to compile properly? The following variables have been used,
> > > ```
> > > export VCPKG_FORCE_SYSTEM_BINARIES=1
> > > export CPU_TARGET="aarch64"
> > > export NUM_THREADS=4
> > > ```
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > My compiling machine is 8C
> >
> >
> > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) I think these are just warnings and based on my current testing, they do not affect our main functions, so I ignored them. These warnings also appears in our CI environment: https://github.com/apache/incubator-gluten/actions/runs/16491898666/job/46628360332#step:5:1799.
> > <img alt="Image" width="1889" height="952" src="https://private-user-images.githubusercontent.com/31196226/470230088-1eb86399-1d6b-4c15-8a1e-6919f9d01032.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTM3Nzc3NTcsIm5iZiI6MTc1Mzc3NzQ1NywicGF0aCI6Ii8zMTE5NjIyNi80NzAyMzAwODgtMWViODYzOTktMWQ2Yi00YzE1LThhMWUtNjkxOWY5ZDAxMDMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzI5VDA4MjQxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhN2M4NGNkM2Y2NmIxOTJmYTQyMzI1NDNlYTFkY2IzOWM1Nzc5MTU1YzE0ZTc3MWU0ZDFlYzMzZjk4ZWZjNmMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.nQLGIcq6Q3Znw3OceowyqSorqSAFG8bQVCbppQTfaXU">
>
> @kevinw66 Okay, thanks for the tip. My ARM machine currently doesn't have enough memory to compile. I also have a question about the minimum memory required for this compilation. I can compile with less than 32GB on an x86 machine using the same image, but there are only 16GB machines for the ARM architecture. Can you give me a more accurate memory reference so I can apply for a machine? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3131319563
> > > > > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, If all CPU cores are used for compilation, ARM is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of `NUM_THREADS`.
> > > >
> > > >
> > > > [@kevinw66](https://github.com/kevinw66) Hello, I have a question. I have used the images of `vcpkg-centos-8, centos-8-jdk11, and 1.4-vcpkg-centos-8`, but there are these errors.
> > > > ```
> > > > [137/647] Building CXX object velox/buffer/CMakeFiles/velox.dir/__/vector/VectorStream.cpp.o
> > > > In file included from /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/xsimd.hpp:55,
> > > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/base/SimdUtil.h:24,
> > > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/RawVector.h:21,
> > > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/Scratch.h:19,
> > > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/ByteStream.h:18,
> > > > from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/VectorStream.h:22,
> > > > from /incubator-gluten/ep/build-velox/build/velox_ep/velox/vector/VectorStream.cpp:16:
> > > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<float, float> xsimd::sincos(float)’:
> > > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:981:54: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> > > > 981 | inline std::pair<float, float> sincos(float val) noexcept
> > > > | ^~~~~~~~
> > > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<double, double> xsimd::sincos(double)’:
> > > > /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:988:57: note: parameter passing for argument of type ‘std::pair<double, double>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> > > > 988 | inline std::pair<double, double> sincos(double val) noexcept
> > > > ```
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > What needs to be adjusted to compile properly? The following variables have been used,
> > > > ```
> > > > export VCPKG_FORCE_SYSTEM_BINARIES=1
> > > > export CPU_TARGET="aarch64"
> > > > export NUM_THREADS=4
> > > > ```
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > My compiling machine is 8C
> > >
> > >
> > > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) I think these are just warnings and based on my current testing, they do not affect our main functions, so I ignored them. These warnings also appears in our CI environment: https://github.com/apache/incubator-gluten/actions/runs/16491898666/job/46628360332#step:5:1799.
> > > <img alt="Image" width="1889" height="952" src="https://private-user-images.githubusercontent.com/31196226/470230088-1eb86399-1d6b-4c15-8a1e-6919f9d01032.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTM3Nzc3NTcsIm5iZiI6MTc1Mzc3NzQ1NywicGF0aCI6Ii8zMTE5NjIyNi80NzAyMzAwODgtMWViODYzOTktMWQ2Yi00YzE1LThhMWUtNjkxOWY5ZDAxMDMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzI5VDA4MjQxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhN2M4NGNkM2Y2NmIxOTJmYTQyMzI1NDNlYTFkY2IzOWM1Nzc5MTU1YzE0ZTc3MWU0ZDFlYzMzZjk4ZWZjNmMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.nQLGIcq6Q3Znw3OceowyqSorqSAFG8bQVCbppQTfaXU">
>
> [@kevinw66](https://github.com/kevinw66) Okay, thanks for the tip. My ARM machine currently doesn't have enough memory to compile. I also have a question about the minimum memory required for this compilation. I can compile with less than 32GB on an x86 machine using the same image, but there are only 16GB machines for the ARM architecture. Can you give me a more accurate memory reference so I can apply for a machine? Thanks!
@A-little-bit-of-data Actually 16GB is enough for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3083051399
> You may use pidstat to check the executor's CPU%, see if there is IO bottleneck
The number of CPU cores configured on both sides is the same. Checking the container platform monitoring, the CPU usage of the pods on both sides is executed according to limit 1, and both have reached the upper limit. According to my understanding, even if there is a CPU bottleneck, it should be faster with gluten. Is there any problem with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3083093146
> which query it is?
>
> If CPU is the bottleneck, Gluten is faster, if IO is bottleneck, GPU CPU% is lower
Two tests were conducted
```
with year_total as (
select c_customer_id customer_id
,c_first_name customer_first_name
,c_last_name customer_last_name
,c_preferred_cust_flag customer_preferred_cust_flag
,c_birth_country customer_birth_country
,c_login customer_login
,c_email_address customer_email_address
,d_year dyear
,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2) year_total
,'s' sale_type
from sf100.customer
,sf100.store_sales
,sf100.date_dim
where c_customer_sk = ss_customer_sk
and ss_sold_date_sk = d_date_sk
group by c_customer_id
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_country
,c_login
,c_email_address
,d_year
union all
select c_customer_id customer_id
,c_first_name customer_first_name
,c_last_name customer_last_name
,c_preferred_cust_flag customer_preferred_cust_flag
,c_birth_country customer_birth_country
,c_login customer_login
,c_email_address customer_email_address
,d_year dyear
,sum((((cs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2) ) year_total
,'c' sale_type
from sf100.customer
,sf100.catalog_sales
,sf100.date_dim
where c_customer_sk = cs_bill_customer_sk
and cs_sold_date_sk = d_date_sk
group by c_customer_id
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_country
,c_login
,c_email_address
,d_year
union all
select c_customer_id customer_id
,c_first_name customer_first_name
,c_last_name customer_last_name
,c_preferred_cust_flag customer_preferred_cust_flag
,c_birth_country customer_birth_country
,c_login customer_login
,c_email_address customer_email_address
,d_year dyear
,sum((((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2) ) year_total
,'w' sale_type
from sf100.customer
,sf100.web_sales
,sf100.date_dim
where c_customer_sk = ws_bill_customer_sk
and ws_sold_date_sk = d_date_sk
group by c_customer_id
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_country
,c_login
,c_email_address
,d_year
)
select t_s_secyear.customer_id
,t_s_secyear.customer_first_name
,t_s_secyear.customer_last_name
,t_s_secyear.customer_birth_country
from year_total t_s_firstyear
,year_total t_s_secyear
,year_total t_c_firstyear
,year_total t_c_secyear
,year_total t_w_firstyear
,year_total t_w_secyear
where t_s_secyear.customer_id = t_s_firstyear.customer_id
and t_s_firstyear.customer_id = t_c_secyear.customer_id
and t_s_firstyear.customer_id = t_c_firstyear.customer_id
and t_s_firstyear.customer_id = t_w_firstyear.customer_id
and t_s_firstyear.customer_id = t_w_secyear.customer_id
and t_s_firstyear.sale_type = 's'
and t_c_firstyear.sale_type = 'c'
and t_w_firstyear.sale_type = 'w'
and t_s_secyear.sale_type = 's'
and t_c_secyear.sale_type = 'c'
and t_w_secyear.sale_type = 'w'
and t_s_firstyear.dyear = 1999
and t_s_secyear.dyear = 1999+1
and t_c_firstyear.dyear = 1999
and t_c_secyear.dyear = 1999+1
and t_w_firstyear.dyear = 1999
and t_w_secyear.dyear = 1999+1
and t_s_firstyear.year_total > 0
and t_c_firstyear.year_total > 0
and t_w_firstyear.year_total > 0
and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end
> case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end
and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end
> case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end
order by t_s_secyear.customer_id
,t_s_secyear.customer_first_name
,t_s_secyear.customer_last_name
,t_s_secyear.customer_birth_country
;
```
```
with wscs as
(select sold_date_sk
,sales_price
from (select ws_sold_date_sk sold_date_sk
,ws_ext_sales_price sales_price
from sf100.web_sales
union all
select cs_sold_date_sk sold_date_sk
,cs_ext_sales_price sales_price
from sf100.catalog_sales)),
wswscs as
(select d_week_seq,
sum(case when (d_day_name='Sunday') then sales_price else null end) sun_sales,
sum(case when (d_day_name='Monday') then sales_price else null end) mon_sales,
sum(case when (d_day_name='Tuesday') then sales_price else null end) tue_sales,
sum(case when (d_day_name='Wednesday') then sales_price else null end) wed_sales,
sum(case when (d_day_name='Thursday') then sales_price else null end) thu_sales,
sum(case when (d_day_name='Friday') then sales_price else null end) fri_sales,
sum(case when (d_day_name='Saturday') then sales_price else null end) sat_sales
from wscs
,sf100.date_dim
where d_date_sk = sold_date_sk
group by d_week_seq)
select d_week_seq1
,round(sun_sales1/sun_sales2,2)
,round(mon_sales1/mon_sales2,2)
,round(tue_sales1/tue_sales2,2)
,round(wed_sales1/wed_sales2,2)
,round(thu_sales1/thu_sales2,2)
,round(fri_sales1/fri_sales2,2)
,round(sat_sales1/sat_sales2,2)
from
(select wswscs.d_week_seq d_week_seq1
,sun_sales sun_sales1
,mon_sales mon_sales1
,tue_sales tue_sales1
,wed_sales wed_sales1
,thu_sales thu_sales1
,fri_sales fri_sales1
,sat_sales sat_sales1
from wswscs,sf100.date_dim
where date_dim.d_week_seq = wswscs.d_week_seq and
d_year = 1998) y,
(select wswscs.d_week_seq d_week_seq2
,sun_sales sun_sales2
,mon_sales mon_sales2
,tue_sales tue_sales2
,wed_sales wed_sales2
,thu_sales thu_sales2
,fri_sales fri_sales2
,sat_sales sat_sales2
from wswscs
,sf100.date_dim
where date_dim.d_week_seq = wswscs.d_week_seq and
d_year = 1998+1) z
where d_week_seq1=d_week_seq2-53
order by d_week_seq1;
```
I am using it on a container platform. Through monitoring, I see that the CPU has reached the configured limit. I don't know if this is considered a bottleneck. I cannot execute pidstat in the pod.
I have compiled it myself, and the size of the compiled jar is about the same as the pre-compiled jar provided by the official. The log contains
```
Component Velox Branch: <unknown>
Component Velox Revision: <unknown>
Component Velox Revision Time: <unknown>
GCC Version: <unknown>
Java Version: <unknown>
```
Could this be the reason why there is no big difference in calculation speed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3111766326
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data), this seems a known issue that has been reported in Gluten weekly build job a few weeks ago: https://github.com/apache/incubator-gluten/actions/runs/16252911851/job/45885046535
>
> [@kevinw66](https://github.com/kevinw66), do you have any update?
@PHILO-HE It's not the same. The weekly job failure is due to a specific patch in openEuler's GCC. This problem does not exist if you use the community version of GCC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3050701652
> Can you try to build in docker? https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md
Hello, I checked the document you mentioned, and both methods in it have official compilation images, which is great! But the spark image is the default Ubuntu image. I looked for the official compilation base images, which are all based on centos. If I use this centos-based image to compile, will there be any impact or problems?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3209542823
Assume this issue has been fixed. Please submit a separate issue if there is any build issue on ARM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3112341051
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, If all CPU cores are used for compilation, ARM is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of `NUM_THREADS`.
Yes, I used it before compiling.Is my error related to the number of CPUs used? I'll try it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3112797043
> > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, If all CPU cores are used for compilation, ARM is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of `NUM_THREADS`.
>
> [@kevinw66](https://github.com/kevinw66) Hello, I have a question. I have used the images of `vcpkg-centos-8, centos-8-jdk11, and 1.4-vcpkg-centos-8`, but there are these errors.
>
> ```
> [137/647] Building CXX object velox/buffer/CMakeFiles/velox.dir/__/vector/VectorStream.cpp.o
> In file included from /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/xsimd.hpp:55,
> from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/base/SimdUtil.h:24,
> from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/RawVector.h:21,
> from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/Scratch.h:19,
> from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/ByteStream.h:18,
> from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/VectorStream.h:22,
> from /incubator-gluten/ep/build-velox/build/velox_ep/velox/vector/VectorStream.cpp:16:
> /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<float, float> xsimd::sincos(float)’:
> /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:981:54: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> 981 | inline std::pair<float, float> sincos(float val) noexcept
> | ^~~~~~~~
> /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<double, double> xsimd::sincos(double)’:
> /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:988:57: note: parameter passing for argument of type ‘std::pair<double, double>’ when C++17 is enabled changed to match C++14 in GCC 10.1
> 988 | inline std::pair<double, double> sincos(double val) noexcept
> ```
>
> What needs to be adjusted to compile properly? The following variables have been used,
>
> ```
> export VCPKG_FORCE_SYSTEM_BINARIES=1
> export CPU_TARGET="aarch64"
> export NUM_THREADS=4
> ```
>
> My compiling machine is 8C
I think these are just warnings and based on my current testing, they do not affect our main functions, so I ignored them. These warnings also appears in our CI environment.
<img width="1889" height="952" alt="Image" src="https://github.com/user-attachments/assets/1eb86399-1d6b-4c15-8a1e-6919f9d01032" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3072124958
> > Thank you very much for your reply. I currently see `https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md`. There are two ways to compile. Which one do you recommend for better compatibility?
>
> If you build on centos7 and use on ubuntu20, you must use static link.
Check the static link written in the document. The image used is `apache/gluten:vcpkg-centos-7`, which is only for amd64 architecture. I used the image of `apache/gluten:1.4-vcpkg-centos-8`, and then used the dockerfile in the document. When building, an error occurred: `/bin/sh: /opt/rh/devtoolset-11/enable: No such file or directory. `The dockerfile used is as follows:
```
FROM apache/gluten:1.4-vcpkg-centos-8
# Build Gluten Jar
RUN source /opt/rh/devtoolset-11/enable && \
git clone https://github.com/apache/incubator-gluten.git && \
cd incubator-gluten && \
./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_vcpkg=ON --build_arrow=ON && \
mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests
```
But I ran a pod using the apache/gluten:1.4-vcpkg-centos-8 image and found that there was
<img width="840" height="199" alt="Image" src="https://github.com/user-attachments/assets/ece5e440-f2d3-46a8-9a01-c267cdcc1cbc" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3071776444
> Yes
Thank you very much for your reply. I currently see `https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md`. There are two ways to compile. Which one do you recommend for better compatibility?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3102077933
> > [@PHILO-HE](https://github.com/PHILO-HE) I would like to ask how to handle this message and display the corresponding version in the log. According to your suggestion, I added the configuration and tested it again, and the effect is still very good. Thank you.
>
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data), great to see it works. I have proposed a patch to fix the missing build information. Could you help verify it on your side? [#10243](https://github.com/apache/incubator-gluten/pull/10243)
@PHILO-HE No problem, but I need an arm architecture image now to compile a package that can be used under the arm architecture. Is there an available image? I have tried `vcpkg-centos-8, centos-8-jdk11, and 1.4-vcpkg-centos-8.` None of them work. The errors reported at the end of the compilation are
```
[460/647] Building CXX object velox/buffer/CMakeFiles/velox.dir/__/functions/prestosql/aggregates/MinByAggregate.cpp.o
In file included from /incubator-gluten/ep/build-velox/build/velox_ep/_build/release/_deps/xsimd-src/include/xsimd/xsimd.hpp:55,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/base/SimdUtil.h:24,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/RawVector.h:21,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/Scratch.h:19,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/ByteStream.h:18,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/HashStringAllocator.h:20,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/functions/lib/aggregates/Compare.h:19,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/functions/prestosql/aggregates/MinMaxByAggregateBase.h:18,
from /incubator-gluten/ep/build-velox/build/velox_ep/velox/functions/prestosql/aggregates/MinByAggregate.cpp:19:
/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/_deps/xsimd-src/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<float, float> xsimd::sincos(float)’:
/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/_deps/xsimd-src/include/xsimd/arch/xsimd_scalar.hpp:981:54: note: parameter passing for argument of type std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
981 | inline std::pair<float, float> sincos(float val) noexcept
| ^~~~~~~~
/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/_deps/xsimd-src/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<double, double> xsimd::sincos(double)’:
/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/_deps/xsimd-src/include/xsimd/arch/xsimd_scalar.hpp:988:57: note: parameter passing for argument of type std::pair<double, double>’ when C++17 is enabled changed to match C++14 in GCC 10.1
988 | inline std::pair<double, double> sincos(double val) noexcept
| ^~~~~~~~
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:105: build] Error 1
make[1]: Leaving directory '/incubator-gluten/ep/build-velox/build/velox_ep'
make: *** [Makefile:112: release] Error 2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3082669822
The same spark3.5.2 image is used and the same configuration is started in k8s
##gluten
___admin___.spark.app.name=admin
___admin___.spark.executor.instances=1
___admin___.spark.driver.cores=1
___admin___.spark.executor.cores=1
___admin___.spark.kubernetes.driver.limit.cores=1
___admin___.spark.kubernetes.executor.limit.cores=1
___admin___.spark.driver.memory=1g
___admin___.spark.executor.memory=1g
___admin___.spark.memory.offHeap.enabled=true
___admin___.spark.memory.offHeap.size=7g
___admin___.spark.dynamicAllocation.enabled=true
___admin___.spark.dynamicAllocation.shuffleTracking.enabled=true
___admin___.spark.dynamicAllocation.schedulerBacklogTimeout=1s
___admin___.spark.dynamicAllocation.sustainedSchedulerBacklogTimeout = 30s
___admin___.spark.dynamicAllocation.minExecutors=1
___admin___.spark.dynamicAllocation.maxExecutors=5
___admin___.spark.dynamicAllocation.initialExecutors=2
##No gluten
___admin___.spark.app.name=admin
___admin___.spark.executor.instances=1
___admin___.spark.driver.cores=1
___admin___.spark.executor.cores=1
___admin___.spark.kubernetes.driver.limit.cores=1
___admin___.spark.kubernetes.executor.limit.cores=1
___admin___.spark.driver.memory=1g
___admin___.spark.executor.memory=8g
___admin___.spark.dynamicAllocation.enabled=true
___admin___.spark.dynamicAllocation.shuffleTracking.enabled=true
___admin___.spark.dynamicAllocation.schedulerBacklogTimeout=1s
___admin___.spark.dynamicAllocation.sustainedSchedulerBacklogTimeout = 30s
___admin___.spark.dynamicAllocation.minExecutors=1
___admin___.spark.dynamicAllocation.maxExecutors=5
___admin___.spark.dynamicAllocation.initialExecutors=2
Executed SQL
```
with year_total as (
select c_customer_id customer_id
,c_first_name customer_first_name
,c_last_name customer_last_name
,c_preferred_cust_flag customer_preferred_cust_flag
,c_birth_country customer_birth_country
,c_login customer_login
,c_email_address customer_email_address
,d_year dyear
,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2) year_total
,'s' sale_type
from sf100.customer
,sf100.store_sales
,sf100.date_dim
where c_customer_sk = ss_customer_sk
and ss_sold_date_sk = d_date_sk
group by c_customer_id
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_country
,c_login
,c_email_address
,d_year
union all
select c_customer_id customer_id
,c_first_name customer_first_name
,c_last_name customer_last_name
,c_preferred_cust_flag customer_preferred_cust_flag
,c_birth_country customer_birth_country
,c_login customer_login
,c_email_address customer_email_address
,d_year dyear
,sum((((cs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2) ) year_total
,'c' sale_type
from sf100.customer
,sf100.catalog_sales
,sf100.date_dim
where c_customer_sk = cs_bill_customer_sk
and cs_sold_date_sk = d_date_sk
group by c_customer_id
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_country
,c_login
,c_email_address
,d_year
union all
select c_customer_id customer_id
,c_first_name customer_first_name
,c_last_name customer_last_name
,c_preferred_cust_flag customer_preferred_cust_flag
,c_birth_country customer_birth_country
,c_login customer_login
,c_email_address customer_email_address
,d_year dyear
,sum((((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2) ) year_total
,'w' sale_type
from sf100.customer
,sf100.web_sales
,sf100.date_dim
where c_customer_sk = ws_bill_customer_sk
and ws_sold_date_sk = d_date_sk
group by c_customer_id
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_country
,c_login
,c_email_address
,d_year
)
select t_s_secyear.customer_id
,t_s_secyear.customer_first_name
,t_s_secyear.customer_last_name
,t_s_secyear.customer_birth_country
from year_total t_s_firstyear
,year_total t_s_secyear
,year_total t_c_firstyear
,year_total t_c_secyear
,year_total t_w_firstyear
,year_total t_w_secyear
where t_s_secyear.customer_id = t_s_firstyear.customer_id
and t_s_firstyear.customer_id = t_c_secyear.customer_id
and t_s_firstyear.customer_id = t_c_firstyear.customer_id
and t_s_firstyear.customer_id = t_w_firstyear.customer_id
and t_s_firstyear.customer_id = t_w_secyear.customer_id
and t_s_firstyear.sale_type = 's'
and t_c_firstyear.sale_type = 'c'
and t_w_firstyear.sale_type = 'w'
and t_s_secyear.sale_type = 's'
and t_c_secyear.sale_type = 'c'
and t_w_secyear.sale_type = 'w'
and t_s_firstyear.dyear = 1999
and t_s_secyear.dyear = 1999+1
and t_c_firstyear.dyear = 1999
and t_c_secyear.dyear = 1999+1
and t_w_firstyear.dyear = 1999
and t_w_secyear.dyear = 1999+1
and t_s_firstyear.year_total > 0
and t_c_firstyear.year_total > 0
and t_w_firstyear.year_total > 0
and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end
> case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end
and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end
> case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end
order by t_s_secyear.customer_id
,t_s_secyear.customer_first_name
,t_s_secyear.customer_last_name
,t_s_secyear.customer_birth_country
;
```
##gluten
<img width="1019" height="570" alt="Image" src="https://github.com/user-attachments/assets/0b90edd3-34e9-4d5f-b694-4fe6e24b07f8" />
##No gluten
<img width="1766" height="1013" alt="Image" src="https://github.com/user-attachments/assets/ac96bdad-1226-4e93-ad34-2c5c6a46c73d" />
The test results did not show the improvement as much as expected. This is still under the sf100 database. When the volume is small, the situation is still upside down.
Of course, I only selected one of the sql tests in tpch, not all of them. Is there something wrong with my use? Please give me some advice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3209538350
> [@kevinw66](https://github.com/kevinw66) OK, thanks for your suggestion. I'll try it later. But why does the official website recommend at least 64GB of RAM? What's the reason for this?
This is a recommended size for efficiently building Gluten + Velox, but it's not a strict requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE closed issue #10102: [VL] Compiling 1.4.0 on X86 fails
URL: https://github.com/apache/incubator-gluten/issues/10102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3034669868
Can you try to build in docker?
https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3031719436
@A-little-bit-of-data, any error reported earlier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3032387189
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data), any error reported earlier?[@A-little-bit-of-data](https://github.com/A-little-bit-of-data) ，之前报告了任何错误吗？
Previously, there were some errors in compiling C++, missing packages and so on. After completing the packages, it got stuck here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3077722885
> > [211/211] Linking CXX executable vcpkg-test Environment variable VCPKG_FORCE_SYSTEM_BINARIES must be set on arm, s390x, ppc64le and riscv platforms. The command '/bin/sh -c source /opt/rh/gcc-toolset-11/enable && git clone -b branch-1.4 [apache/incubator-gluten.git](https://github.com/apache/incubator-gluten.git) && cd incubator-gluten && ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_vcpkg=ON --build_arrow=ON && mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests' returned a non-zero code: 1
> > How to verify this Environment variable VCPKG_FORCE_SYSTEM_BINARIES? I compiled it on the arm architecture.I used the image of apache/gluten:1.4-vcpkg-centos-8
>
> Oh, centos-7 doesn't support ARM. centos8 support but we never tried to use the image build from centos8 on ubuntu20. you may try
I switched to X86 architecture to compile, and the compiled dockerfile is:
```
FROM apache/gluten:vcpkg-centos-7
# Build Gluten Jar
RUN source /opt/rh/devtoolset-11/enable && \
git clone -b branch-1.4 https://github.com/apache/incubator-gluten.git && \
cd incubator-gluten && \
./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_vcpkg=ON --build_arrow=ON && \
mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests
```
After compiling, put /incubator-gluten/package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar in the spark jars directory, then start the cluster to view the logs.
```
25/07/16 17:13:18 INFO Discovery: Start discovering components in the current classpath...
25/07/16 17:13:18 INFO Discovery: Discovered component files: org.apache.gluten.backendsapi.velox.VeloxBackend. Duration: 70 ms.
25/07/16 17:13:18 INFO package: Components registered within order: velox
25/07/16 17:13:18 INFO GlutenDriverPlugin: Gluten build info:
==============================================================
Components: Velox
Component Velox Branch: <unknown>
Component Velox Revision: <unknown>
Component Velox Revision Time: <unknown>
Gluten Version: 1.4.0
GCC Version: <unknown>
Java Version: <unknown>
Scala Version: 2.12.15
Spark Version: 3.5.2
Hadoop Version: 3.3.4
Gluten Branch: branch-1.4
Gluten Revision: 50dd117dadb9fe3e9bba5c5002168e1294b2e2bb
Gluten Revision Time: 2025-06-06 15:51:55 +0800
Gluten Build Time: 2025-07-16T06:16:11Z
Gluten Repo URL: https://hk.gh-proxy.com/https://github.com/apache/incubator-gluten.git
==============================================================
25/07/16 17:13:18 INFO VeloxListenerApi: Memory overhead is not set. Setting it to 1288490188 automatically. Gluten doesn't follow Spark's calculation on default value of this option because the actual required memory overhead will depend on off-heap usage than on on-heap usage.
25/07/16 17:13:18 INFO SparkDirectoryUtil: Created local directory at /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836
25/07/16 17:13:18 INFO JniWorkspace: Creating JNI workspace in root directory /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7
25/07/16 17:13:18 INFO JniWorkspace: JNI workspace /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754 created in root directory /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7
25/07/16 17:13:19 INFO JniLibLoader: Read real path /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libgluten.so for libPath /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libgluten.so
25/07/16 17:13:19 INFO JniLibLoader: Library /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libgluten.so has been loaded using path-loading method
25/07/16 17:13:19 INFO JniLibLoader: Library /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libgluten.so has been loaded
25/07/16 17:13:19 INFO JniLibLoader: Successfully loaded library linux/amd64/libgluten.so
25/07/16 17:13:20 INFO JniLibLoader: Read real path /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libvelox.so for libPath /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libvelox.so
25/07/16 17:13:20 INFO JniLibLoader: Library /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libvelox.so has been loaded using path-loading method
25/07/16 17:13:20 INFO JniLibLoader: Library /var/data/spark-f59074c0-3c5f-488a-b29c-679e62ab1a59/gluten-b1b9b1ad-c680-4632-8175-e35f87099836/jni/85f2aba6-e18b-4925-830a-9a21acc801b7/gluten-2942424804660574754/linux/amd64/libvelox.so has been loaded
25/07/16 17:13:20 INFO JniLibLoader: Successfully loaded library linux/amd64/libvelox.so
W20250716 17:13:20.953872 31 MemoryArbitrator.cpp:84] Query memory capacity[921.00MB] is set for NOOP arbitrator which has no capacity enforcement
25/07/16 17:13:20 INFO DriverPluginContainer: Initialized driver component for plugin org.apache.gluten.GlutenPlugin.
```
Why does this happen:
```
Component Velox Branch: <unknown>
Component Velox Revision: <unknown>
Component Velox Revision Time: <unknown>
GCC Version: <unknown>
Java Version: <unknown>
```
In the compiled container,
ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
java -version
openjdk version "1.8.0_412"
OpenJDK Runtime Environment (build 1.8.0_412-b08)
OpenJDK 64-Bit Server VM (build 25.412-b08, mixed mode)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3071767485
> > > Can you try to build in docker? https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md
> >
> >
> > Hello, I checked the document you mentioned, and both methods in it have official compilation images, which is great! But the spark image is the default Ubuntu image. I looked for the official compilation base image, which is based on centos. If I use this centos-based image to compile, will there be any impact or problems? Then I am using the kirin operating system of the arm architecture. Can this machine be applied to the above compilation method?
>
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data), Gluten doesn't require that the build time OS must be consistent with runtime OS. But glibc compatibility should be guaranteed. And the CPU architecture, whether ARM or x86, should remain consistent between the build and runtime environments. In our CI, we build Gluten on centos-7 with static link, and run some test jobs on ubuntu-20.04/22.04, centos-8.
glibc compatibility should be guaranteed. I checked the document and found the supported spark3.5.2 version. I pulled the spark3.5.2 image and checked the version. The results are as follows:
```
ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.16) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
```
I checked `https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md` and `https://hub.docker.com/r/apache/gluten/tags ` I found that they are all centos related images, but not ubuntu. Can I directly use, for example, apache/gluten:vcpkg-centos-7 image to compile, and then use it in spark3.5.2 image?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3095015236
> > [@A-little-bit-of-data](https://github.com/A-little-bit-of-data), the `<unknown>` message in the log may not be related to the perf. issue.
> > Are the queries you tested standard TPC-DS queries? If so, the result is inconsistent with ours. Please check Spark history server's web UI to see whether unexpected operator fallback happens, which can cause perf. degradation. BTW, have you enabled Gluten columnar shuffle in your test?
>
> [@PHILO-HE](https://github.com/PHILO-HE) My query test statement comes from `https://www.cnblogs.com/syw20170419/p/16012807.html;` I used the default configuration for gluten and did not configure it separately.
@PHILO-HE I would like to ask how to handle this <unknown> message and display the corresponding version in the log.
According to your suggestion, I added the configuration and tested it again, and the effect is still very good. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3072475512
> > Thank you very much for your reply. I currently see `https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md`. There are two ways to compile. Which one do you recommend for better compatibility?
>
> If you build on centos7 and use on ubuntu20, you must use static link.
[211/211] Linking CXX executable vcpkg-test
Environment variable VCPKG_FORCE_SYSTEM_BINARIES must be set on arm, s390x, ppc64le and riscv platforms.
The command '/bin/sh -c source /opt/rh/gcc-toolset-11/enable && git clone -b branch-1.4 https://github.com/apache/incubator-gluten.git && cd incubator-gluten && ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_vcpkg=ON --build_arrow=ON && mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests' returned a non-zero code: 1
How to verify this Environment variable VCPKG_FORCE_SYSTEM_BINARIES? I compiled it on the arm architecture.I used the image of apache/gluten:1.4-vcpkg-centos-8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3071772159
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3132701271
@kevinw66 OK, thanks for your suggestion. I'll try it later. But why does the official website recommend at least 64GB of RAM? What's the reason for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3106651071
@A-little-bit-of-data, this seems a known issue that has been reported in Gluten weekly build job a few weeks ago: https://github.com/apache/incubator-gluten/actions/runs/16252911851/job/45885046535
@kevinw66, do you have any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3083013244
You may use pidstat to check the executor's CPU%, see if there is IO bottleneck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3088419629
@A-little-bit-of-data, the `<unknown>` message in the log may not be related to the perf. issue.
Are the queries you tested standard TPC-DS queries? If so, the result is inconsistent with ours. Please check Spark history server's web UI to see whether unexpected operator fallback happens, which can cause perf. degradation. BTW, have you enabled Gluten columnar shuffle in your test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3101465108
> [@PHILO-HE](https://github.com/PHILO-HE) I would like to ask how to handle this message and display the corresponding version in the log. According to your suggestion, I added the configuration and tested it again, and the effect is still very good. Thank you.
@A-little-bit-of-data, great to see it works. I have proposed a patch to fix the missing build information. Could you help verify it on your side? https://github.com/apache/incubator-gluten/pull/10243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3094971476
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data), the `<unknown>` message in the log may not be related to the perf. issue.
>
> Are the queries you tested standard TPC-DS queries? If so, the result is inconsistent with ours. Please check Spark history server's web UI to see whether unexpected operator fallback happens, which can cause perf. degradation. BTW, have you enabled Gluten columnar shuffle in your test?
My query test statement comes from `https://www.cnblogs.com/syw20170419/p/16012807.html;` I used the default configuration for gluten and did not configure it separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3112739909
> [@A-little-bit-of-data](https://github.com/A-little-bit-of-data) Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, If all CPU cores are used for compilation, ARM is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of `NUM_THREADS`.
@kevinw66 Hello, I have a question. I have used the images of `vcpkg-centos-8, centos-8-jdk11, and 1.4-vcpkg-centos-8`, but there are these errors.
```
[137/647] Building CXX object velox/buffer/CMakeFiles/velox.dir/__/vector/VectorStream.cpp.o
In file included from /incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/xsimd.hpp:55,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/base/SimdUtil.h:24,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/RawVector.h:21,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/Scratch.h:19,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/common/memory/ByteStream.h:18,
from /incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/VectorStream.h:22,
from /incubator-gluten/ep/build-velox/build/velox_ep/velox/vector/VectorStream.cpp:16:
/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<float, float> xsimd::sincos(float)’:
/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:981:54: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
981 | inline std::pair<float, float> sincos(float val) noexcept
| ^~~~~~~~
/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp: In function ‘std::pair<double, double> xsimd::sincos(double)’:
/incubator-gluten/dev/vcpkg/vcpkg_installed/arm64-linux-release/include/xsimd/arch/xsimd_scalar.hpp:988:57: note: parameter passing for argument of type ‘std::pair<double, double>’ when C++17 is enabled changed to match C++14 in GCC 10.1
988 | inline std::pair<double, double> sincos(double val) noexcept
```
What needs to be adjusted to compile properly? The following variables have been used,
```
export VCPKG_FORCE_SYSTEM_BINARIES=1
export CPU_TARGET="aarch64"
export NUM_THREADS=4
```
My compiling machine is 8C
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3111752907
@A-little-bit-of-data Have you tried export `CPU_TARGET="aarch64"` and `VCPKG_FORCE_SYSTEM_BINARIES=1`? Also, under the same resource conditions, ARM compilation is more likely to crash than x86. The reason is still unclear for now, but you can try reducing the value of NUM_THREADS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3083066567
which query it is?
If CPU is the bottleneck, Gluten is faster, if IO is bottleneck, GPU CPU% is lower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3072643883
> [211/211] Linking CXX executable vcpkg-test Environment variable VCPKG_FORCE_SYSTEM_BINARIES must be set on arm, s390x, ppc64le and riscv platforms. The command '/bin/sh -c source /opt/rh/gcc-toolset-11/enable && git clone -b branch-1.4 [apache/incubator-gluten.git](https://github.com/apache/incubator-gluten.git) && cd incubator-gluten && ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_vcpkg=ON --build_arrow=ON && mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests' returned a non-zero code: 1
>
> How to verify this Environment variable VCPKG_FORCE_SYSTEM_BINARIES? I compiled it on the arm architecture.I used the image of apache/gluten:1.4-vcpkg-centos-8
Oh, centos-7 doesn't support ARM. centos8 support but we never tried to use the image build from centos8 on ubuntu20. you may try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Compiling 1.4.0 on X86 fails [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #10102:
URL: https://github.com/apache/incubator-gluten/issues/10102#issuecomment-3050828985
> > Can you try to build in docker? https://github.com/apache/incubator-gluten/blob/main/docs/developers/velox-backend-build-in-docker.md
>
> Hello, I checked the document you mentioned, and both methods in it have official compilation images, which is great! But the spark image is the default Ubuntu image. I looked for the official compilation base image, which is based on centos. If I use this centos-based image to compile, will there be any impact or problems? Then I am using the kirin operating system of the arm architecture. Can this machine be applied to the above compilation method?
@A-little-bit-of-data, Gluten doesn't require that the build time OS must be consistent with runtime OS. But glibc compatibility should be guaranteed. And the CPU architecture, whether ARM or x86, should remain consistent between the build and runtime environments. In our CI, we build Gluten on centos-7 with static link, and run some test jobs on ubuntu-20.04/22.04, centos-8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org