You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by ta...@apache.org on 2026/07/01 11:30:31 UTC
(gluten) branch fake_add_bolt_backend created (now fe46c376c5)
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a change to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
at fe46c376c5 add bolt backend in gluten
This branch includes the following new commits:
new b6c7ba6c08 [GLUTEN][CORE] Pass Spark task attempt id and pool name from Java to native runtime/memory-manager
new cdbbb62a88 [GLUTEN][VL] Support sequence function in Velox backend
new ab54bf2a81 [GLUTEN][CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers
new 28a3a9e733 [GLUTEN][CORE] Defer literal node construction for InSet to reduce memory
new f9aa6e0ffa [GLUTEN][VL] Support format_number function in Velox backend
new 91483bcf41 [GLUTEN][CORE] Expose ColumnarBatches.isLightBatch/ensureOffloaded as public
new 7bbd377877 [GLUTEN][CORE] Support HiveGenericUDTF in HiveUDFTransformer
new b12003c1f1 [GLUTEN][CORE] Move overwrite sql-tests to a backend-neutral shared dir
new aa18e3ba22 [GLUTEN][CORE] Move benchmark data to shared directory
new 90e224a833 [GLUTEN][CORE] Add backend hook for sort aggregate offload
new b59a38602f [GLUTEN][CORE] Add backend hook for sequence expressions
new 1ae8a16664 [GLUTEN][CORE] Expose shuffle reader metrics iterator delegate
new 33107d98a1 [GLUTEN][CORE] Add common Paimon scan hooks and metadata plumbing
new bbaf1c218e [GLUTEN][CORE] Extract shared write-file metrics updater
new 50bd8f717f [GLUTEN][CORE] Share array-to-string cast rewrite rule
new 9b911c4618 [GLUTEN][CORE] Move TopN transformer into backends-core
new 3d280271e0 [GLUTEN][CORE] Move ColumnarBatchSerializerInstance to backends-core
new 992ce0407f [GLUTEN][CORE] Share Iceberg visitor and proto helpers
new 5ae169adc9 [GLUTEN][CORE] Extract shared Celeborn factory bases
new 151f2420dc [GLUTEN][CORE] Share native write test checker base
new 91d67dc30d [GLUTEN][CORE] Extract shared Substrait extension registry
new e04ac1d151 [GLUTEN][CORE] Extract more shared cpp helper headers into cpp/core
new abe3f09666 [GLUTEN][CORE] Extract shared VariantToVectorConverter and Jol filesystem registration into cpp/core
new 9b1f9f828f [GLUTEN][CORE] Share JVM test input resources
new f89483359a [GLUTEN][CORE] Share JVM tier-1 test scaffolding under backends-core
new 3d4e02631a [GLUTEN][CORE] Share WholeStageTransformerSuite under backends-core
new a3b3e94d53 [GLUTEN][CORE] Share tier-2 test suites under backends-core
new beea58c2e8 [GLUTEN][CORE] Share OperatorMetrics and Metrics POJOs in backends-core
new fda31ae5c4 [GLUTEN][CORE] Share metrics updaters in backends-core
new f9f85ee0f4 [GLUTEN][CORE] Share RowToColumnarExec under backends-core
new e66e23e365 [GLUTEN][CORE] Switch shared RowToColumnarFuzzer to SharedRowToColumnarExec
new 0302293fa4 [GLUTEN][CORE] Share JNI wrapper classes under backends-core
new 26ec3497b8 [GLUTEN][CORE] Share tier-2 function test suites under backends-core
new 3bc6243683 [GLUTEN][CORE] Share ColumnarCollectTailExec under backends-core
new a938159153 Share carrier row transitions under backends-core
new 94eb1d5628 [GLUTEN][CORE] Share velox execution helpers in backends-core
new ee2f50bad6 [GLUTEN][CORE] Share additional tier-2 test suites
new fe46c376c5 add bolt backend in gluten
The 38 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 19/38: [GLUTEN][CORE] Extract shared Celeborn factory bases
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 5ae169adc978041f66848457c04cbd8ff37aa7ea
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:49:45 2026 +0800
[GLUTEN][CORE] Extract shared Celeborn factory bases
Move the duplicated Celeborn serializer and shuffle writer factory scaffolding into backends-core so backend modules only keep the backend-specific class name and writer construction logic.
---
.../CHCelebornColumnarBatchSerializerFactory.scala | 10 +++-----
.../CHCelebornColumnarShuffleWriterFactory.scala | 5 ++--
backends-core/pom.xml | 30 ++++++++++++++++++++++
...redCelebornColumnarBatchSerializerFactory.scala | 6 ++---
...haredCelebornColumnarShuffleWriterFactory.scala | 20 ++++++++-------
...loxCelebornColumnarBatchSerializerFactory.scala | 10 +++-----
...VeloxCelebornColumnarShuffleWriterFactory.scala | 5 ++--
7 files changed, 54 insertions(+), 32 deletions(-)
diff --git a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala b/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala
index 8eb40c0c7f..11b11eb0bc 100644
--- a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala
+++ b/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala
@@ -16,10 +16,6 @@
*/
package org.apache.spark.shuffle
-import org.apache.spark.shuffle.gluten.celeborn.CelebornColumnarBatchSerializerFactory
-
-class CHCelebornColumnarBatchSerializerFactory extends CelebornColumnarBatchSerializerFactory {
-
- override def columnarBatchSerializerClass(): String =
- "org.apache.spark.shuffle.CHCelebornColumnarBatchSerializer"
-}
+class CHCelebornColumnarBatchSerializerFactory
+ extends SharedCelebornColumnarBatchSerializerFactory(
+ "org.apache.spark.shuffle.CHCelebornColumnarBatchSerializer")
diff --git a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala b/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala
index d8393b0932..d81d1cd21f 100644
--- a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala
+++ b/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala
@@ -18,14 +18,13 @@ package org.apache.spark.shuffle
import org.apache.spark.TaskContext
import org.apache.spark.shuffle.celeborn.CelebornShuffleHandle
-import org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleWriterFactory
import org.apache.celeborn.client.ShuffleClient
import org.apache.celeborn.common.CelebornConf
-class CHCelebornColumnarShuffleWriterFactory extends CelebornShuffleWriterFactory {
+class CHCelebornColumnarShuffleWriterFactory extends SharedCelebornColumnarShuffleWriterFactory {
- override def createShuffleWriterInstance[K, V](
+ override protected def createBackendShuffleWriter[K, V](
shuffleId: Int,
handle: CelebornShuffleHandle[K, V, V],
context: TaskContext,
diff --git a/backends-core/pom.xml b/backends-core/pom.xml
index 113cbed416..8903c9a45a 100644
--- a/backends-core/pom.xml
+++ b/backends-core/pom.xml
@@ -94,6 +94,36 @@
</build>
<profiles>
+ <profile>
+ <id>celeborn</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-celeborn</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.celeborn</groupId>
+ <artifactId>celeborn-client-spark-${spark.major.version}-shaded_${scala.binary.version}</artifactId>
+ <version>${celeborn.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.celeborn</groupId>
+ <artifactId>celeborn-client-spark-${spark.major.version}_${scala.binary.version}</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.celeborn</groupId>
+ <artifactId>celeborn-spark-${spark.major.version}-columnar-shuffle_${scala.binary.version}</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
<profile>
<id>iceberg</id>
<activation>
diff --git a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala b/backends-core/src-celeborn/main/scala/org/apache/spark/shuffle/SharedCelebornColumnarBatchSerializerFactory.scala
similarity index 80%
copy from backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala
copy to backends-core/src-celeborn/main/scala/org/apache/spark/shuffle/SharedCelebornColumnarBatchSerializerFactory.scala
index 8eb40c0c7f..d11f2d3d42 100644
--- a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarBatchSerializerFactory.scala
+++ b/backends-core/src-celeborn/main/scala/org/apache/spark/shuffle/SharedCelebornColumnarBatchSerializerFactory.scala
@@ -18,8 +18,8 @@ package org.apache.spark.shuffle
import org.apache.spark.shuffle.gluten.celeborn.CelebornColumnarBatchSerializerFactory
-class CHCelebornColumnarBatchSerializerFactory extends CelebornColumnarBatchSerializerFactory {
+abstract class SharedCelebornColumnarBatchSerializerFactory(serializerClassName: String)
+ extends CelebornColumnarBatchSerializerFactory {
- override def columnarBatchSerializerClass(): String =
- "org.apache.spark.shuffle.CHCelebornColumnarBatchSerializer"
+ final override def columnarBatchSerializerClass(): String = serializerClassName
}
diff --git a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala b/backends-core/src-celeborn/main/scala/org/apache/spark/shuffle/SharedCelebornColumnarShuffleWriterFactory.scala
similarity index 71%
copy from backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala
copy to backends-core/src-celeborn/main/scala/org/apache/spark/shuffle/SharedCelebornColumnarShuffleWriterFactory.scala
index d8393b0932..d76bdca5c3 100644
--- a/backends-clickhouse/src-celeborn/main/scala/org/apache/spark/shuffle/CHCelebornColumnarShuffleWriterFactory.scala
+++ b/backends-core/src-celeborn/main/scala/org/apache/spark/shuffle/SharedCelebornColumnarShuffleWriterFactory.scala
@@ -23,21 +23,23 @@ import org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleWriterFactory
import org.apache.celeborn.client.ShuffleClient
import org.apache.celeborn.common.CelebornConf
-class CHCelebornColumnarShuffleWriterFactory extends CelebornShuffleWriterFactory {
+abstract class SharedCelebornColumnarShuffleWriterFactory extends CelebornShuffleWriterFactory {
- override def createShuffleWriterInstance[K, V](
+ protected def createBackendShuffleWriter[K, V](
+ shuffleId: Int,
+ handle: CelebornShuffleHandle[K, V, V],
+ context: TaskContext,
+ celebornConf: CelebornConf,
+ client: ShuffleClient,
+ writeMetrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V]
+
+ final override def createShuffleWriterInstance[K, V](
shuffleId: Int,
handle: CelebornShuffleHandle[K, V, V],
context: TaskContext,
celebornConf: CelebornConf,
client: ShuffleClient,
writeMetrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = {
- new CHCelebornColumnarShuffleWriter[K, V](
- shuffleId,
- handle,
- context,
- celebornConf,
- client,
- writeMetrics)
+ createBackendShuffleWriter(shuffleId, handle, context, celebornConf, client, writeMetrics)
}
}
diff --git a/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarBatchSerializerFactory.scala b/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarBatchSerializerFactory.scala
index f763d392ac..726e192310 100644
--- a/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarBatchSerializerFactory.scala
+++ b/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarBatchSerializerFactory.scala
@@ -16,10 +16,6 @@
*/
package org.apache.spark.shuffle
-import org.apache.spark.shuffle.gluten.celeborn.CelebornColumnarBatchSerializerFactory
-
-class VeloxCelebornColumnarBatchSerializerFactory extends CelebornColumnarBatchSerializerFactory {
-
- override def columnarBatchSerializerClass(): String =
- "org.apache.spark.shuffle.CelebornColumnarBatchSerializer"
-}
+class VeloxCelebornColumnarBatchSerializerFactory
+ extends SharedCelebornColumnarBatchSerializerFactory(
+ "org.apache.spark.shuffle.CelebornColumnarBatchSerializer")
diff --git a/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriterFactory.scala b/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriterFactory.scala
index e09d601ab9..7b975dc6b1 100644
--- a/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriterFactory.scala
+++ b/backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriterFactory.scala
@@ -18,14 +18,13 @@ package org.apache.spark.shuffle
import org.apache.spark.TaskContext
import org.apache.spark.shuffle.celeborn.CelebornShuffleHandle
-import org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleWriterFactory
import org.apache.celeborn.client.ShuffleClient
import org.apache.celeborn.common.CelebornConf
-class VeloxCelebornColumnarShuffleWriterFactory extends CelebornShuffleWriterFactory {
+class VeloxCelebornColumnarShuffleWriterFactory extends SharedCelebornColumnarShuffleWriterFactory {
- override def createShuffleWriterInstance[K, V](
+ override protected def createBackendShuffleWriter[K, V](
shuffleId: Int,
handle: CelebornShuffleHandle[K, V, V],
context: TaskContext,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 16/38: [GLUTEN][CORE] Move TopN transformer into backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 9b911c46180fa674ec300ec27f63a6aef6ac89b6
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:14:25 2026 +0800
[GLUTEN][CORE] Move TopN transformer into backends-core
Relocate the reusable TopN Substrait execution glue into backends-core while keeping its package and call sites unchanged so backend-specific planning can continue to reference the shared transformer transparently.
---
.../src/main/scala/org/apache/gluten/execution/TopNTransformer.scala | 0
1 file changed, 0 insertions(+), 0 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/TopNTransformer.scala b/backends-core/src/main/scala/org/apache/gluten/execution/TopNTransformer.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/execution/TopNTransformer.scala
rename to backends-core/src/main/scala/org/apache/gluten/execution/TopNTransformer.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 01/38: [GLUTEN][CORE] Pass Spark task attempt id and pool name from Java to native runtime/memory-manager
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit b6c7ba6c080f985401b4180344e1aea893e42167
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Fri Jun 26 15:35:15 2026 +0800
[GLUTEN][CORE] Pass Spark task attempt id and pool name from Java to native runtime/memory-manager
This patch extends RuntimeJniWrapper.createRuntime and the NativeMemoryManagerJniWrapper
create/hold/release JNI methods with extra parameters that propagate the Spark task
attempt id (and a memory-pool name) from the JVM down to the native side.
Unlike the previous getter/setter approach, the task attempt id is now passed through
the Runtime constructor/factory chain (and the pool name through the MemoryManager
constructor/factory chain), so backends receive the value at construction time. The
native ThreadManager wiring from the community is preserved. No behavior change for
existing Velox backends; this unblocks future backends (e.g. Bolt) that need
task-level identification at the native layer with minimal rebase conflicts.
Generated-by: Claude claude-sonnet-4-6
---
cpp/core/compute/Runtime.cc | 5 +--
cpp/core/compute/Runtime.h | 20 +++++++++---
cpp/core/jni/JniWrapper.cc | 38 ++++++++++++++--------
cpp/core/memory/MemoryManager.cc | 5 +--
cpp/core/memory/MemoryManager.h | 13 ++++++--
cpp/velox/compute/VeloxBackend.cc | 12 ++++---
cpp/velox/compute/VeloxRuntime.cc | 5 +--
cpp/velox/compute/VeloxRuntime.h | 3 +-
cpp/velox/memory/VeloxMemoryManager.cc | 5 +--
cpp/velox/memory/VeloxMemoryManager.h | 3 +-
.../memory/NativeMemoryManagerJniWrapper.java | 6 ++--
.../apache/gluten/runtime/RuntimeJniWrapper.java | 2 +-
.../apache/gluten/memory/NativeMemoryManager.scala | 9 +++--
.../scala/org/apache/gluten/runtime/Runtime.scala | 4 ++-
14 files changed, 88 insertions(+), 42 deletions(-)
diff --git a/cpp/core/compute/Runtime.cc b/cpp/core/compute/Runtime.cc
index f283bc7283..83e724b28a 100644
--- a/cpp/core/compute/Runtime.cc
+++ b/cpp/core/compute/Runtime.cc
@@ -41,9 +41,10 @@ Runtime* Runtime::create(
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& sessionConf) {
+ const std::unordered_map<std::string, std::string>& sessionConf,
+ int64_t taskId) {
auto& factory = runtimeFactories().get(kind);
- return factory(kind, memoryManager, threadManager, sessionConf);
+ return factory(kind, memoryManager, threadManager, sessionConf, taskId);
}
void Runtime::release(Runtime* runtime) {
diff --git a/cpp/core/compute/Runtime.h b/cpp/core/compute/Runtime.h
index 07120684d3..ae9ce0c56e 100644
--- a/cpp/core/compute/Runtime.h
+++ b/cpp/core/compute/Runtime.h
@@ -68,14 +68,16 @@ class Runtime : public std::enable_shared_from_this<Runtime> {
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& sessionConf)>;
+ const std::unordered_map<std::string, std::string>& sessionConf,
+ int64_t taskId)>;
using Releaser = std::function<void(Runtime*)>;
static void registerFactory(const std::string& kind, Factory factory, Releaser releaser);
static Runtime* create(
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& sessionConf = {});
+ const std::unordered_map<std::string, std::string>& sessionConf = {},
+ int64_t taskId = -1);
static void release(Runtime*);
static std::optional<std::string>* localWriteFilesTempPath();
static std::optional<std::string>* localWriteFileName();
@@ -84,8 +86,13 @@ class Runtime : public std::enable_shared_from_this<Runtime> {
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& confMap)
- : kind_(kind), memoryManager_(memoryManager), threadManager_(threadManager), confMap_(confMap) {}
+ const std::unordered_map<std::string, std::string>& confMap,
+ int64_t taskId = -1)
+ : kind_(kind),
+ memoryManager_(memoryManager),
+ threadManager_(threadManager),
+ confMap_(confMap),
+ taskId_(taskId) {}
virtual ~Runtime() = default;
@@ -194,6 +201,10 @@ class Runtime : public std::enable_shared_from_this<Runtime> {
return objStore_->save(obj);
}
+ int64_t taskId() const {
+ return taskId_;
+ }
+
protected:
std::string kind_;
MemoryManager* memoryManager_;
@@ -206,5 +217,6 @@ class Runtime : public std::enable_shared_from_this<Runtime> {
std::optional<SparkTaskInfo> taskInfo_{std::nullopt};
std::shared_ptr<WholeStageDumper> dumper_{nullptr};
+ int64_t taskId_{-1};
};
} // namespace gluten
diff --git a/cpp/core/jni/JniWrapper.cc b/cpp/core/jni/JniWrapper.cc
index efdfd36cba..af01d92ca0 100644
--- a/cpp/core/jni/JniWrapper.cc
+++ b/cpp/core/jni/JniWrapper.cc
@@ -156,7 +156,7 @@ inline static const std::string kInternalBackendKind{"internal"};
class InternalMemoryManager : public MemoryManager {
public:
- InternalMemoryManager(const std::string& kind) : MemoryManager(kind) {}
+ InternalMemoryManager(const std::string& kind, const std::string& name) : MemoryManager(kind, name) {}
arrow::MemoryPool* defaultArrowMemoryPool() override {
throw GlutenException("Not implemented");
@@ -183,12 +183,16 @@ class InternalRuntime : public Runtime {
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& confMap)
- : Runtime(kind, memoryManager, threadManager, confMap) {}
+ const std::unordered_map<std::string, std::string>& confMap,
+ int64_t taskId)
+ : Runtime(kind, memoryManager, threadManager, confMap, taskId) {}
};
-MemoryManager* internalMemoryManagerFactory(const std::string& kind, std::unique_ptr<AllocationListener> listener) {
- return new InternalMemoryManager(kind);
+MemoryManager* internalMemoryManagerFactory(
+ const std::string& kind,
+ std::unique_ptr<AllocationListener> listener,
+ const std::string& name) {
+ return new InternalMemoryManager(kind, name);
}
void internalMemoryManagerReleaser(MemoryManager* memoryManager) {
@@ -220,8 +224,9 @@ Runtime* internalRuntimeFactory(
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& sessionConf) {
- return new InternalRuntime(kind, memoryManager, threadManager, sessionConf);
+ const std::unordered_map<std::string, std::string>& sessionConf,
+ int64_t taskId) {
+ return new InternalRuntime(kind, memoryManager, threadManager, sessionConf, taskId);
}
void internalRuntimeReleaser(Runtime* runtime) {
@@ -345,7 +350,8 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_runtime_RuntimeJniWrapper_createR
jstring jBackendType,
jlong nmmHandle,
jlong ntmHandle,
- jbyteArray sessionConf) {
+ jbyteArray sessionConf,
+ jlong taskAttemptId) {
JNI_METHOD_START
MemoryManager* memoryManager = jniCastOrThrow<MemoryManager>(nmmHandle);
ThreadManager* threadManager = jniCastOrThrow<ThreadManager>(ntmHandle);
@@ -353,7 +359,8 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_runtime_RuntimeJniWrapper_createR
auto sparkConf = parseConfMap(env, safeArray.elems(), safeArray.length());
auto backendType = jStringToCString(env, jBackendType);
- auto runtime = Runtime::create(backendType, memoryManager, threadManager, sparkConf);
+ auto runtime =
+ Runtime::create(backendType, memoryManager, threadManager, sparkConf, static_cast<int64_t>(taskAttemptId));
return reinterpret_cast<jlong>(runtime);
JNI_METHOD_END(kInvalidObjectHandle)
}
@@ -378,13 +385,15 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_memory_NativeMemoryManagerJniWrap
jclass,
jstring jBackendType,
jobject jListener,
- jbyteArray sessionConf) {
+ jbyteArray sessionConf,
+ jstring jName) {
JNI_METHOD_START
JavaVM* vm;
if (env->GetJavaVM(&vm) != JNI_OK) {
throw GlutenException("Unable to get JavaVM instance");
}
auto backendType = jStringToCString(env, jBackendType);
+ auto name = jStringToCString(env, jName);
auto safeArray = getByteArrayElementsSafe(env, sessionConf);
auto sparkConf = parseConfMap(env, safeArray.elems(), safeArray.length());
std::unique_ptr<AllocationListener> listener = std::make_unique<SparkAllocationListener>(vm, jListener);
@@ -392,7 +401,7 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_memory_NativeMemoryManagerJniWrap
if (backtrace) {
listener = std::make_unique<BacktraceAllocationListener>(std::move(listener));
}
- MemoryManager* mm = MemoryManager::create(backendType, std::move(listener));
+ MemoryManager* mm = MemoryManager::create(backendType, std::move(listener), name);
return reinterpret_cast<jlong>(mm);
JNI_METHOD_END(-1L)
}
@@ -457,7 +466,9 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_memory_NativeMemoryManagerJniWrap
JNIEXPORT void JNICALL Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold( // NOLINT
JNIEnv* env,
jclass,
- jlong nmmHandle) {
+ jlong nmmHandle,
+ jstring jName,
+ jlong taskAttemptId) {
JNI_METHOD_START
auto* memoryManager = jniCastOrThrow<MemoryManager>(nmmHandle);
memoryManager->hold();
@@ -467,7 +478,8 @@ JNIEXPORT void JNICALL Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapp
JNIEXPORT void JNICALL Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release( // NOLINT
JNIEnv* env,
jclass,
- jlong nmmHandle) {
+ jlong nmmHandle,
+ jlong taskAttemptId) {
JNI_METHOD_START
auto* memoryManager = jniCastOrThrow<MemoryManager>(nmmHandle);
MemoryManager::release(memoryManager);
diff --git a/cpp/core/memory/MemoryManager.cc b/cpp/core/memory/MemoryManager.cc
index 8ccf8af2cc..07984c51d8 100644
--- a/cpp/core/memory/MemoryManager.cc
+++ b/cpp/core/memory/MemoryManager.cc
@@ -35,9 +35,10 @@ void MemoryManager::registerFactory(const std::string& kind, MemoryManager::Fact
memoryManagerReleasers().registerObj(kind, std::move(releaser));
}
-MemoryManager* MemoryManager::create(const std::string& kind, std::unique_ptr<AllocationListener> listener) {
+MemoryManager*
+MemoryManager::create(const std::string& kind, std::unique_ptr<AllocationListener> listener, const std::string& name) {
auto& factory = memoryManagerFactories().get(kind);
- return factory(kind, std::move(listener));
+ return factory(kind, std::move(listener), name);
}
void MemoryManager::release(MemoryManager* memoryManager) {
diff --git a/cpp/core/memory/MemoryManager.h b/cpp/core/memory/MemoryManager.h
index 93d875ad1e..9b003086b9 100644
--- a/cpp/core/memory/MemoryManager.h
+++ b/cpp/core/memory/MemoryManager.h
@@ -25,13 +25,15 @@ namespace gluten {
class MemoryManager {
public:
- using Factory = std::function<MemoryManager*(const std::string& kind, std::unique_ptr<AllocationListener> listener)>;
+ using Factory = std::function<
+ MemoryManager*(const std::string& kind, std::unique_ptr<AllocationListener> listener, const std::string& name)>;
using Releaser = std::function<void(MemoryManager*)>;
static void registerFactory(const std::string& kind, Factory factory, Releaser releaser);
- static MemoryManager* create(const std::string& kind, std::unique_ptr<AllocationListener> listener);
+ static MemoryManager*
+ create(const std::string& kind, std::unique_ptr<AllocationListener> listener, const std::string& name = "");
static void release(MemoryManager*);
- MemoryManager(const std::string& kind) : kind_(kind){};
+ MemoryManager(const std::string& kind, const std::string& name = "") : kind_(kind), name_(name){};
virtual ~MemoryManager() = default;
@@ -39,6 +41,10 @@ class MemoryManager {
return kind_;
}
+ std::string name() const {
+ return name_;
+ }
+
// Get the default Arrow memory pool for this memory manager. This memory pool is held by the memory manager.
virtual arrow::MemoryPool* defaultArrowMemoryPool() = 0;
@@ -58,6 +64,7 @@ class MemoryManager {
private:
std::string kind_;
+ std::string name_;
};
} // namespace gluten
diff --git a/cpp/velox/compute/VeloxBackend.cc b/cpp/velox/compute/VeloxBackend.cc
index aa77d0ffe7..8bdd3fac0f 100644
--- a/cpp/velox/compute/VeloxBackend.cc
+++ b/cpp/velox/compute/VeloxBackend.cc
@@ -77,8 +77,11 @@ using namespace facebook;
namespace gluten {
namespace {
-MemoryManager* veloxMemoryManagerFactory(const std::string& kind, std::unique_ptr<AllocationListener> listener) {
- return new VeloxMemoryManager(kind, std::move(listener), *VeloxBackend::get()->getBackendConf());
+MemoryManager* veloxMemoryManagerFactory(
+ const std::string& kind,
+ std::unique_ptr<AllocationListener> listener,
+ const std::string& name) {
+ return new VeloxMemoryManager(kind, std::move(listener), *VeloxBackend::get()->getBackendConf(), name);
}
void veloxMemoryManagerReleaser(MemoryManager* memoryManager) {
@@ -89,10 +92,11 @@ Runtime* veloxRuntimeFactory(
const std::string& kind,
MemoryManager* memoryManager,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& sessionConf) {
+ const std::unordered_map<std::string, std::string>& sessionConf,
+ int64_t taskId) {
auto* vmm = dynamic_cast<VeloxMemoryManager*>(memoryManager);
GLUTEN_CHECK(vmm != nullptr, "Not a Velox memory manager");
- return new VeloxRuntime(kind, vmm, threadManager, sessionConf);
+ return new VeloxRuntime(kind, vmm, threadManager, sessionConf, taskId);
}
void veloxRuntimeReleaser(Runtime* runtime) {
diff --git a/cpp/velox/compute/VeloxRuntime.cc b/cpp/velox/compute/VeloxRuntime.cc
index f13430bd0c..a9155eee4b 100644
--- a/cpp/velox/compute/VeloxRuntime.cc
+++ b/cpp/velox/compute/VeloxRuntime.cc
@@ -238,8 +238,9 @@ VeloxRuntime::VeloxRuntime(
const std::string& kind,
VeloxMemoryManager* vmm,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& confMap)
- : Runtime(kind, vmm, threadManager, confMap) {
+ const std::unordered_map<std::string, std::string>& confMap,
+ int64_t taskId)
+ : Runtime(kind, vmm, threadManager, confMap, taskId) {
// Refresh session config.
veloxCfg_ =
std::make_shared<facebook::velox::config::ConfigBase>(std::unordered_map<std::string, std::string>(confMap_));
diff --git a/cpp/velox/compute/VeloxRuntime.h b/cpp/velox/compute/VeloxRuntime.h
index c6ee1c462c..5e04f254cd 100644
--- a/cpp/velox/compute/VeloxRuntime.h
+++ b/cpp/velox/compute/VeloxRuntime.h
@@ -43,7 +43,8 @@ class VeloxRuntime final : public Runtime {
const std::string& kind,
VeloxMemoryManager* vmm,
ThreadManager* threadManager,
- const std::unordered_map<std::string, std::string>& confMap);
+ const std::unordered_map<std::string, std::string>& confMap,
+ int64_t taskId = -1);
~VeloxRuntime() override;
diff --git a/cpp/velox/memory/VeloxMemoryManager.cc b/cpp/velox/memory/VeloxMemoryManager.cc
index ad46a6ce9b..1bfd487b69 100644
--- a/cpp/velox/memory/VeloxMemoryManager.cc
+++ b/cpp/velox/memory/VeloxMemoryManager.cc
@@ -221,8 +221,9 @@ ArbitratorFactoryRegister::~ArbitratorFactoryRegister() {
VeloxMemoryManager::VeloxMemoryManager(
const std::string& kind,
std::unique_ptr<AllocationListener> listener,
- const facebook::velox::config::ConfigBase& backendConf)
- : MemoryManager(kind), listener_(std::move(listener)) {
+ const facebook::velox::config::ConfigBase& backendConf,
+ const std::string& name)
+ : MemoryManager(kind, name), listener_(std::move(listener)) {
auto reservationBlockSize =
backendConf.get<uint64_t>(kMemoryReservationBlockSize, kMemoryReservationBlockSizeDefault);
blockListener_ = std::make_unique<BlockAllocationListener>(listener_.get(), reservationBlockSize);
diff --git a/cpp/velox/memory/VeloxMemoryManager.h b/cpp/velox/memory/VeloxMemoryManager.h
index 9d4b9e9a9b..216ce88b8b 100644
--- a/cpp/velox/memory/VeloxMemoryManager.h
+++ b/cpp/velox/memory/VeloxMemoryManager.h
@@ -59,7 +59,8 @@ class VeloxMemoryManager final : public MemoryManager {
VeloxMemoryManager(
const std::string& kind,
std::unique_ptr<AllocationListener> listener,
- const facebook::velox::config::ConfigBase& backendConf);
+ const facebook::velox::config::ConfigBase& backendConf,
+ const std::string& name = "");
~VeloxMemoryManager() override;
VeloxMemoryManager(const VeloxMemoryManager&) = delete;
diff --git a/gluten-arrow/src/main/java/org/apache/gluten/memory/NativeMemoryManagerJniWrapper.java b/gluten-arrow/src/main/java/org/apache/gluten/memory/NativeMemoryManagerJniWrapper.java
index c23b9704ec..15dc8f3152 100644
--- a/gluten-arrow/src/main/java/org/apache/gluten/memory/NativeMemoryManagerJniWrapper.java
+++ b/gluten-arrow/src/main/java/org/apache/gluten/memory/NativeMemoryManagerJniWrapper.java
@@ -22,13 +22,13 @@ public class NativeMemoryManagerJniWrapper {
private NativeMemoryManagerJniWrapper() {}
public static native long create(
- String backendType, ReservationListener listener, byte[] sessionConf);
+ String backendType, ReservationListener listener, byte[] sessionConf, String name);
public static native byte[] collectUsage(long handle);
public static native long shrink(long handle, long size);
- public static native void hold(long handle);
+ public static native void hold(long handle, String name, long taskAttemptId);
- public static native void release(long handle);
+ public static native void release(long handle, long taskAttemptId);
}
diff --git a/gluten-arrow/src/main/java/org/apache/gluten/runtime/RuntimeJniWrapper.java b/gluten-arrow/src/main/java/org/apache/gluten/runtime/RuntimeJniWrapper.java
index ed22759672..cb22250e4e 100644
--- a/gluten-arrow/src/main/java/org/apache/gluten/runtime/RuntimeJniWrapper.java
+++ b/gluten-arrow/src/main/java/org/apache/gluten/runtime/RuntimeJniWrapper.java
@@ -21,7 +21,7 @@ public class RuntimeJniWrapper {
private RuntimeJniWrapper() {}
public static native long createRuntime(
- String backendType, long nmm, long ntm, byte[] sessionConf);
+ String backendType, long nmm, long ntm, byte[] sessionConf, long taskAttemptId);
public static native void releaseRuntime(long handle);
}
diff --git a/gluten-arrow/src/main/scala/org/apache/gluten/memory/NativeMemoryManager.scala b/gluten-arrow/src/main/scala/org/apache/gluten/memory/NativeMemoryManager.scala
index 159e1bba5e..ed3fba8ee3 100644
--- a/gluten-arrow/src/main/scala/org/apache/gluten/memory/NativeMemoryManager.scala
+++ b/gluten-arrow/src/main/scala/org/apache/gluten/memory/NativeMemoryManager.scala
@@ -23,6 +23,7 @@ import org.apache.gluten.memory.memtarget.{KnownNameAndStats, MemoryTarget, Spil
import org.apache.gluten.proto.MemoryUsageStats
import org.apache.gluten.utils.ConfigUtil
+import org.apache.spark.TaskContext
import org.apache.spark.memory.SparkMemoryUtil
import org.apache.spark.sql.internal.{GlutenConfigUtil, SQLConf}
import org.apache.spark.task.{TaskResource, TaskResources}
@@ -54,7 +55,8 @@ object NativeMemoryManager {
ConfigUtil.serialize(
GlutenConfig
.getNativeSessionConf(backendName, GlutenConfigUtil.parseConfig(SQLConf.get.getAllConfs))
- .asJava)
+ .asJava),
+ name
)
spillers.append(new Spiller() {
override def spill(self: MemoryTarget, phase: Spiller.Phase, size: Long): Long = phase match {
@@ -76,7 +78,8 @@ object NativeMemoryManager {
private val released: AtomicBoolean = new AtomicBoolean(false)
override def addSpiller(spiller: Spiller): Unit = spillers.append(spiller)
- override def hold(): Unit = NativeMemoryManagerJniWrapper.hold(handle)
+ override def hold(): Unit =
+ NativeMemoryManagerJniWrapper.hold(handle, name, TaskContext.get().taskAttemptId())
override def getHandle(): Long = handle
override def release(): Unit = {
if (!released.compareAndSet(false, true)) {
@@ -97,7 +100,7 @@ object NativeMemoryManager {
LOGGER.debug("About to release memory manager, " + dump())
}
- NativeMemoryManagerJniWrapper.release(handle)
+ NativeMemoryManagerJniWrapper.release(handle, TaskContext.get().taskAttemptId())
if (rl.getUsedBytes != 0) {
LOGGER.warn(
diff --git a/gluten-arrow/src/main/scala/org/apache/gluten/runtime/Runtime.scala b/gluten-arrow/src/main/scala/org/apache/gluten/runtime/Runtime.scala
index 2e3c468140..18b7589de2 100644
--- a/gluten-arrow/src/main/scala/org/apache/gluten/runtime/Runtime.scala
+++ b/gluten-arrow/src/main/scala/org/apache/gluten/runtime/Runtime.scala
@@ -22,6 +22,7 @@ import org.apache.gluten.memory.NativeMemoryManager
import org.apache.gluten.threads.{NativeThreadManager, TaskChildThreadInitializer}
import org.apache.gluten.utils.ConfigUtil
+import org.apache.spark.TaskContext
import org.apache.spark.sql.internal.{GlutenConfigUtil, SQLConf}
import org.apache.spark.task.{TaskResource, TaskResources}
@@ -64,7 +65,8 @@ object Runtime {
(GlutenConfig
.getNativeSessionConf(
backendName,
- GlutenConfigUtil.parseConfig(SQLConf.get.getAllConfs)) ++ extraConf.asScala).asJava)
+ GlutenConfigUtil.parseConfig(SQLConf.get.getAllConfs)) ++ extraConf.asScala).asJava),
+ TaskContext.get().taskAttemptId()
)
private val released: AtomicBoolean = new AtomicBoolean(false)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 30/38: [GLUTEN][CORE] Share RowToColumnarExec under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit f9f85ee0f466c0df35e24041ec84b34efebaf777
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 21:02:51 2026 +0800
[GLUTEN][CORE] Share RowToColumnarExec under backends-core
Extract the common Row -> Arrow ColumnarBatch conversion logic from
RowToVeloxColumnarExec into a new SharedRowToColumnarExec under
backends-core. Backends only need to subclass it and supply two
hooks:
- preferredBatchBytes (the backend's preferred Arrow batch byte size)
- sparkToBackendUnsafe (the backend's `BroadcastUtils.sparkToXxxUnsafe`)
VeloxRowToColumnarExec becomes a thin subclass that fills the two
backend-specific hooks. The `object RowToVeloxColumnarExec` keeps
existing call sites by forwarding to SharedRowToColumnarExec.
Generated-by: TraeCli openrouter-3o
---
.../gluten/execution/SharedRowToColumnarExec.scala | 42 ++--
.../gluten/execution/RowToVeloxColumnarExec.scala | 224 +++------------------
2 files changed, 52 insertions(+), 214 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala b/backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala
similarity index 87%
copy from backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala
copy to backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala
index f11de5894a..5f375f53ae 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala
@@ -18,7 +18,7 @@ package org.apache.gluten.execution
import org.apache.gluten.backendsapi.BackendsApiManager
import org.apache.gluten.columnarbatch.ColumnarBatches
-import org.apache.gluten.config.{GlutenConfig, VeloxConfig}
+import org.apache.gluten.config.GlutenConfig
import org.apache.gluten.iterator.Iterators
import org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators
import org.apache.gluten.runtime.Runtimes
@@ -43,19 +43,39 @@ import org.apache.arrow.memory.ArrowBuf
import scala.collection.mutable.ListBuffer
-case class RowToVeloxColumnarExec(child: SparkPlan) extends RowToColumnarExecBase(child = child) {
+/**
+ * Backend-agnostic implementation of `Row -> ColumnarBatch` conversion. Backends only need to
+ * subclass it and supply the per-backend preferred batch byte size and the broadcast unsafe
+ * relation builder.
+ */
+abstract class SharedRowToColumnarExec(child: SparkPlan) extends RowToColumnarExecBase(child) {
+
+ /** Backend-specific preferred batch byte size used to flush an Arrow batch. */
+ protected def preferredBatchBytes: Long
+
+ /**
+ * Backend-specific `Spark -> Unsafe Arrow` broadcast builder. Equivalent to
+ * `BroadcastUtils.sparkTo{Bolt,Velox}Unsafe`.
+ */
+ protected def sparkToBackendUnsafe[F, T](
+ sc: org.apache.spark.SparkContext,
+ mode: org.apache.spark.sql.catalyst.plans.physical.BroadcastMode,
+ schema: StructType,
+ relation: Broadcast[F],
+ itrTransformer: Iterator[InternalRow] => Iterator[ColumnarBatch]): Broadcast[T]
+
override def doExecuteColumnarInternal(): RDD[ColumnarBatch] = {
val numInputRows = longMetric("numInputRows")
val numOutputBatches = longMetric("numOutputBatches")
val convertTime = longMetric("convertTime")
val numRows = GlutenConfig.get.maxBatchSize
- val numBytes = VeloxConfig.get.veloxPreferredBatchBytes
+ val numBytes = preferredBatchBytes
// This avoids calling `schema` in the RDD closure, so that we don't need to include the entire
// plan (this) in the closure.
val localSchema = schema
child.execute().mapPartitions {
rowIterator =>
- RowToVeloxColumnarExec.toColumnarBatchIterator(
+ SharedRowToColumnarExec.toColumnarBatchIterator(
rowIterator,
localSchema,
numInputRows,
@@ -71,16 +91,16 @@ case class RowToVeloxColumnarExec(child: SparkPlan) extends RowToColumnarExecBas
val numOutputBatches = longMetric("numOutputBatches")
val convertTime = longMetric("convertTime")
val numRows = GlutenConfig.get.maxBatchSize
- val numBytes = VeloxConfig.get.veloxPreferredBatchBytes
+ val numBytes = preferredBatchBytes
val mode = BroadcastUtils.getBroadcastMode(outputPartitioning)
val relation = child.executeBroadcast()
- BroadcastUtils.sparkToVeloxUnsafe(
+ sparkToBackendUnsafe(
sparkContext,
mode,
schema,
relation,
itr =>
- RowToVeloxColumnarExec.toColumnarBatchIterator(
+ SharedRowToColumnarExec.toColumnarBatchIterator(
itr,
schema,
numInputRows,
@@ -90,13 +110,9 @@ case class RowToVeloxColumnarExec(child: SparkPlan) extends RowToColumnarExecBas
numBytes)
)
}
-
- // For spark 3.2.
- protected def withNewChildInternal(newChild: SparkPlan): RowToVeloxColumnarExec =
- copy(child = newChild)
}
-object RowToVeloxColumnarExec {
+object SharedRowToColumnarExec {
def toColumnarBatchIterator(
it: Iterator[InternalRow],
@@ -106,7 +122,7 @@ object RowToVeloxColumnarExec {
val numInputRows = new SQLMetric("numInputRows")
val numOutputBatches = new SQLMetric("numOutputBatches")
val convertTime = new SQLMetric("convertTime")
- RowToVeloxColumnarExec.toColumnarBatchIterator(
+ toColumnarBatchIterator(
it,
schema,
numInputRows,
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala b/backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala
index f11de5894a..c8300e8d36 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala
@@ -16,79 +16,28 @@
*/
package org.apache.gluten.execution
-import org.apache.gluten.backendsapi.BackendsApiManager
-import org.apache.gluten.columnarbatch.ColumnarBatches
-import org.apache.gluten.config.{GlutenConfig, VeloxConfig}
-import org.apache.gluten.iterator.Iterators
-import org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators
-import org.apache.gluten.runtime.Runtimes
-import org.apache.gluten.utils.ArrowAbiUtil
-import org.apache.gluten.vectorized._
+import org.apache.gluten.config.VeloxConfig
+import org.apache.spark.SparkContext
import org.apache.spark.broadcast.Broadcast
-import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.{UnsafeProjection, UnsafeRow}
+import org.apache.spark.sql.catalyst.plans.physical.BroadcastMode
import org.apache.spark.sql.execution.{BroadcastUtils, SparkPlan}
import org.apache.spark.sql.execution.metric.SQLMetric
-import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.utils.SparkArrowUtil
import org.apache.spark.sql.vectorized.ColumnarBatch
-import org.apache.spark.task.TaskResources
-import org.apache.spark.unsafe.Platform
-import org.apache.arrow.c.ArrowSchema
-import org.apache.arrow.memory.ArrowBuf
+case class RowToVeloxColumnarExec(child: SparkPlan) extends SharedRowToColumnarExec(child) {
-import scala.collection.mutable.ListBuffer
+ override protected def preferredBatchBytes: Long = VeloxConfig.get.veloxPreferredBatchBytes
-case class RowToVeloxColumnarExec(child: SparkPlan) extends RowToColumnarExecBase(child = child) {
- override def doExecuteColumnarInternal(): RDD[ColumnarBatch] = {
- val numInputRows = longMetric("numInputRows")
- val numOutputBatches = longMetric("numOutputBatches")
- val convertTime = longMetric("convertTime")
- val numRows = GlutenConfig.get.maxBatchSize
- val numBytes = VeloxConfig.get.veloxPreferredBatchBytes
- // This avoids calling `schema` in the RDD closure, so that we don't need to include the entire
- // plan (this) in the closure.
- val localSchema = schema
- child.execute().mapPartitions {
- rowIterator =>
- RowToVeloxColumnarExec.toColumnarBatchIterator(
- rowIterator,
- localSchema,
- numInputRows,
- numOutputBatches,
- convertTime,
- numRows,
- numBytes)
- }
- }
-
- override def doExecuteBroadcast[T](): Broadcast[T] = {
- val numInputRows = longMetric("numInputRows")
- val numOutputBatches = longMetric("numOutputBatches")
- val convertTime = longMetric("convertTime")
- val numRows = GlutenConfig.get.maxBatchSize
- val numBytes = VeloxConfig.get.veloxPreferredBatchBytes
- val mode = BroadcastUtils.getBroadcastMode(outputPartitioning)
- val relation = child.executeBroadcast()
- BroadcastUtils.sparkToVeloxUnsafe(
- sparkContext,
- mode,
- schema,
- relation,
- itr =>
- RowToVeloxColumnarExec.toColumnarBatchIterator(
- itr,
- schema,
- numInputRows,
- numOutputBatches,
- convertTime,
- numRows,
- numBytes)
- )
+ override protected def sparkToBackendUnsafe[F, T](
+ sc: SparkContext,
+ mode: BroadcastMode,
+ schema: StructType,
+ relation: Broadcast[F],
+ itrTransformer: Iterator[InternalRow] => Iterator[ColumnarBatch]): Broadcast[T] = {
+ BroadcastUtils.sparkToVeloxUnsafe(sc, mode, schema, relation, itrTransformer)
}
// For spark 3.2.
@@ -102,19 +51,8 @@ object RowToVeloxColumnarExec {
it: Iterator[InternalRow],
schema: StructType,
columnBatchSize: Int,
- columnBatchBytes: Long): Iterator[ColumnarBatch] = {
- val numInputRows = new SQLMetric("numInputRows")
- val numOutputBatches = new SQLMetric("numOutputBatches")
- val convertTime = new SQLMetric("convertTime")
- RowToVeloxColumnarExec.toColumnarBatchIterator(
- it,
- schema,
- numInputRows,
- numOutputBatches,
- convertTime,
- columnBatchSize,
- columnBatchBytes)
- }
+ columnBatchBytes: Long): Iterator[ColumnarBatch] =
+ SharedRowToColumnarExec.toColumnarBatchIterator(it, schema, columnBatchSize, columnBatchBytes)
def toColumnarBatchIterator(
it: Iterator[InternalRow],
@@ -123,129 +61,13 @@ object RowToVeloxColumnarExec {
numOutputBatches: SQLMetric,
convertTime: SQLMetric,
columnBatchSize: Int,
- columnBatchBytes: Long): Iterator[ColumnarBatch] = {
- if (it.isEmpty) {
- return Iterator.empty
- }
-
- val arrowSchema =
- SparkArrowUtil.toArrowSchema(schema, SQLConf.get.sessionLocalTimeZone)
- val runtime = Runtimes.contextInstance(BackendsApiManager.getBackendName, "RowToColumnar")
- val jniWrapper = NativeRowToColumnarJniWrapper.create(runtime)
- val arrowAllocator = ArrowBufferAllocators.contextInstance()
- val cSchema = ArrowSchema.allocateNew(arrowAllocator)
- val factory = UnsafeProjection
- val converter = factory.create(schema)
- val r2cHandle =
- try {
- ArrowAbiUtil.exportSchema(arrowAllocator, arrowSchema, cSchema)
- jniWrapper.init(cSchema.memoryAddress())
- } finally {
- cSchema.close()
- }
-
- val res: Iterator[ColumnarBatch] = new Iterator[ColumnarBatch] {
- var finished = false
-
- override def hasNext: Boolean = {
- if (finished) {
- false
- } else {
- it.hasNext
- }
- }
-
- def convertToUnsafeRow(row: InternalRow): UnsafeRow = {
- row match {
- case unsafeRow: UnsafeRow => unsafeRow
- case _ =>
- converter.apply(row)
- }
- }
-
- override def next(): ColumnarBatch = {
- var arrowBuf: ArrowBuf = null
- TaskResources.addRecycler("RowToColumnar_arrowBuf", 100) {
- if (arrowBuf != null && arrowBuf.refCnt() != 0) {
- arrowBuf.close()
- }
- }
- val rowLength = new ListBuffer[Long]()
- var rowCount = 0
- var offset = 0L
- while (rowCount < columnBatchSize && offset < columnBatchBytes && !finished) {
- if (!it.hasNext) {
- finished = true
- } else {
- val row = it.next()
- val start = System.currentTimeMillis()
- val unsafeRow = convertToUnsafeRow(row)
- val sizeInBytes = unsafeRow.getSizeInBytes
-
- // allocate buffer based on first row
- if (rowCount == 0) {
- // allocate buffer based on 1st row, but if first row is very big, this will cause OOM
- // maybe we should optimize to list ArrayBuf to native to avoid buf close and allocate
- // 31760L origins from BaseVariableWidthVector.lastValueAllocationSizeInBytes
- // experimental value
- val estimatedBufSize = Math.min(
- Math.max(
- Math.min(sizeInBytes.toDouble * columnBatchSize * 1.2, 31760L * columnBatchSize),
- sizeInBytes.toDouble * 10),
- // Limit the size of the buffer to columnBatchBytes or the size of the first row,
- // whichever is greater so we always have enough space for the first row.
- Math.max(columnBatchBytes, sizeInBytes)
- )
- arrowBuf = arrowAllocator.buffer(estimatedBufSize.toLong)
- }
-
- if ((offset + sizeInBytes) > arrowBuf.capacity()) {
- val bufSize = if (offset + sizeInBytes > columnBatchBytes) {
- // If adding the current row causes the batch size to exceed columnBatchBytes add
- // just enough space to add the current row.
- offset + sizeInBytes
- } else {
- Math.min((offset + sizeInBytes * 2), columnBatchBytes)
- }
- val tmpBuf = arrowAllocator.buffer(bufSize)
- tmpBuf.setBytes(0, arrowBuf, 0, offset)
- arrowBuf.close()
- arrowBuf = tmpBuf
- }
- Platform.copyMemory(
- unsafeRow.getBaseObject,
- unsafeRow.getBaseOffset,
- null,
- arrowBuf.memoryAddress() + offset,
- sizeInBytes)
- offset += sizeInBytes
- rowLength += sizeInBytes.toLong
- rowCount += 1
- convertTime += System.currentTimeMillis() - start
- }
- }
- numInputRows += rowCount
- numOutputBatches += 1
- val startNative = System.currentTimeMillis()
- try {
- val handle = jniWrapper
- .nativeConvertRowToColumnar(r2cHandle, rowLength.toArray, arrowBuf.memoryAddress())
- val cb = ColumnarBatches.create(handle)
- convertTime += System.currentTimeMillis() - startNative
- cb
- } finally {
- arrowBuf.close()
- arrowBuf = null
- }
- }
- }
- Iterators
- .wrap(res)
- .protectInvocationFlow()
- .recycleIterator {
- jniWrapper.close(r2cHandle)
- }
- .recyclePayload(_.close())
- .create()
- }
+ columnBatchBytes: Long): Iterator[ColumnarBatch] =
+ SharedRowToColumnarExec.toColumnarBatchIterator(
+ it,
+ schema,
+ numInputRows,
+ numOutputBatches,
+ convertTime,
+ columnBatchSize,
+ columnBatchBytes)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 22/38: [GLUTEN][CORE] Extract more shared cpp helper headers into cpp/core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit e04ac1d151ce26f78ff80a2e40abe5c9d41985ae
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:49:18 2026 +0800
[GLUTEN][CORE] Extract more shared cpp helper headers into cpp/core
Move another low-risk batch of identical Bolt/Velox helper definitions into
cpp/core, covering UDF/UDAF ABI headers, function registration helpers, the
shared datasource abstraction, common test helper headers, JSON-to-protobuf
parsing helpers, cudf validator declaration and shared JNI UDF glue.
Keep the Velox-side files as thin wrappers or aliases so existing include paths
and backend-specific entry points stay unchanged while future Bolt rebases stop
conflicting on the duplicated bodies.
---
cpp/core/CMakeLists.txt | 1 +
cpp/{velox => core}/cudf/CudfPlanValidator.h | 2 -
cpp/{velox => core}/jni/JniUdf.cc | 16 +++--
cpp/{velox => core}/jni/JniUdf.h | 6 +-
.../operators/functions/Arithmetic.h | 3 +
.../operators/functions/RegistrationAllFunctions.h | 0
.../operators/writer/GlutenDataSource.h} | 10 ++-
cpp/{velox => core}/tests/FilePathGenerator.h | 0
cpp/{velox => core}/tests/JsonToProtoConverter.cc | 18 ++---
.../utils => core/tests}/JsonToProtoConverter.h | 2 +
.../tests/utils/TestAllocationListener.h | 3 +-
cpp/{velox => core}/tests/utils/TestStreamReader.h | 3 +
cpp/{velox => core}/tests/utils/TestUtils.h | 3 +
cpp/{velox => core}/udf/Udaf.h | 0
cpp/{velox => core}/udf/Udf.h | 0
cpp/{velox => core}/utils/JsonToProtoConverter.h | 2 +
cpp/velox/cudf/CudfPlanValidator.h | 11 +---
cpp/velox/jni/JniUdf.cc | 76 ++--------------------
cpp/velox/jni/JniUdf.h | 6 +-
cpp/velox/operators/functions/Arithmetic.h | 51 +--------------
.../operators/functions/RegistrationAllFunctions.h | 6 +-
cpp/velox/operators/writer/VeloxDataSource.h | 26 +-------
cpp/velox/tests/FilePathGenerator.h | 7 +-
cpp/velox/tests/JsonToProtoConverter.cc | 20 +-----
cpp/velox/tests/JsonToProtoConverter.h | 8 +--
cpp/velox/tests/utils/TestAllocationListener.h | 48 +-------------
cpp/velox/tests/utils/TestStreamReader.h | 19 +-----
cpp/velox/tests/utils/TestUtils.h | 8 +--
cpp/velox/udf/Udaf.h | 25 +------
cpp/velox/udf/Udf.h | 24 +------
cpp/velox/utils/JsonToProtoConverter.h | 8 +--
31 files changed, 67 insertions(+), 345 deletions(-)
diff --git a/cpp/core/CMakeLists.txt b/cpp/core/CMakeLists.txt
index d976e92bd0..936bccaa2e 100644
--- a/cpp/core/CMakeLists.txt
+++ b/cpp/core/CMakeLists.txt
@@ -124,6 +124,7 @@ set(SPARK_COLUMNAR_PLUGIN_SRCS
substrait/SubstraitExtensionRegistry.cc
config/GlutenConfig.cc
jni/JniWrapper.cc
+ jni/JniUdf.cc
memory/AllocationListener.cc
memory/MemoryAllocator.cc
memory/MemoryManager.cc
diff --git a/cpp/velox/cudf/CudfPlanValidator.h b/cpp/core/cudf/CudfPlanValidator.h
similarity index 88%
copy from cpp/velox/cudf/CudfPlanValidator.h
copy to cpp/core/cudf/CudfPlanValidator.h
index de4fb150e0..25a30b60b9 100644
--- a/cpp/velox/cudf/CudfPlanValidator.h
+++ b/cpp/core/cudf/CudfPlanValidator.h
@@ -17,13 +17,11 @@
#pragma once
-#include "memory/VeloxMemoryManager.h"
#include "substrait/plan.pb.h"
namespace gluten {
class CudfPlanValidator {
public:
- // Validate if the plan contains cudf unsupported operator except TableScan and ValueStream.
static bool validate(const ::substrait::Plan& substraitPlan);
};
} // namespace gluten
diff --git a/cpp/velox/jni/JniUdf.cc b/cpp/core/jni/JniUdf.cc
similarity index 94%
copy from cpp/velox/jni/JniUdf.cc
copy to cpp/core/jni/JniUdf.cc
index 32514e7ae7..ff9a8f3b31 100644
--- a/cpp/velox/jni/JniUdf.cc
+++ b/cpp/core/jni/JniUdf.cc
@@ -16,6 +16,9 @@
*/
#include "JniUdf.h"
+
+#include <string>
+
#include "jni/JniCommon.h"
#include "udf/UdfLoader.h"
#include "utils/Exception.h"
@@ -32,24 +35,23 @@ static jmethodID registerUDAFMethod;
} // namespace
-void gluten::initVeloxJniUDF(JNIEnv* env) {
+namespace gluten {
+
+void initJniUDF(JNIEnv* env) {
if (env->GetJavaVM(&vm) != JNI_OK) {
throw gluten::GlutenException("Unable to get JavaVM instance");
}
- // classes
udfResolverClass = createGlobalClassReferenceOrError(env, kUdfResolverClassPath.c_str());
-
- // methods
registerUDFMethod = getMethodIdOrError(env, udfResolverClass, "registerUDF", "(Ljava/lang/String;[B[BZZ)V");
registerUDAFMethod = getMethodIdOrError(env, udfResolverClass, "registerUDAF", "(Ljava/lang/String;[B[B[BZZ)V");
}
-void gluten::finalizeVeloxJniUDF(JNIEnv* env) {
+void finalizeJniUDF(JNIEnv* env) {
env->DeleteGlobalRef(udfResolverClass);
}
-void gluten::jniRegisterFunctionSignatures(JNIEnv* env) {
+void jniRegisterFunctionSignatures(JNIEnv* env) {
auto udfLoader = gluten::UdfLoader::getInstance();
const auto& signatures = udfLoader->getRegisteredUdfSignatures();
@@ -92,3 +94,5 @@ void gluten::jniRegisterFunctionSignatures(JNIEnv* env) {
checkException(env);
}
}
+
+} // namespace gluten
diff --git a/cpp/velox/jni/JniUdf.h b/cpp/core/jni/JniUdf.h
similarity index 88%
copy from cpp/velox/jni/JniUdf.h
copy to cpp/core/jni/JniUdf.h
index 568439d184..dbc72c4865 100644
--- a/cpp/velox/jni/JniUdf.h
+++ b/cpp/core/jni/JniUdf.h
@@ -18,14 +18,12 @@
#pragma once
#include <jni.h>
-#include <string>
-#include <unordered_map>
namespace gluten {
-void initVeloxJniUDF(JNIEnv* env);
+void initJniUDF(JNIEnv* env);
-void finalizeVeloxJniUDF(JNIEnv* env);
+void finalizeJniUDF(JNIEnv* env);
void jniRegisterFunctionSignatures(JNIEnv* env);
diff --git a/cpp/velox/operators/functions/Arithmetic.h b/cpp/core/operators/functions/Arithmetic.h
similarity index 99%
copy from cpp/velox/operators/functions/Arithmetic.h
copy to cpp/core/operators/functions/Arithmetic.h
index 9b2e22ac41..c9487c109b 100644
--- a/cpp/velox/operators/functions/Arithmetic.h
+++ b/cpp/core/operators/functions/Arithmetic.h
@@ -14,6 +14,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
+#pragma once
+
#include <folly/CPortability.h>
#include <stdint.h>
#include <cmath>
diff --git a/cpp/velox/operators/functions/RegistrationAllFunctions.h b/cpp/core/operators/functions/RegistrationAllFunctions.h
similarity index 100%
copy from cpp/velox/operators/functions/RegistrationAllFunctions.h
copy to cpp/core/operators/functions/RegistrationAllFunctions.h
diff --git a/cpp/velox/operators/writer/VeloxDataSource.h b/cpp/core/operators/writer/GlutenDataSource.h
similarity index 87%
copy from cpp/velox/operators/writer/VeloxDataSource.h
copy to cpp/core/operators/writer/GlutenDataSource.h
index 043f1b8b17..74ad1f180d 100644
--- a/cpp/velox/operators/writer/VeloxDataSource.h
+++ b/cpp/core/operators/writer/GlutenDataSource.h
@@ -17,6 +17,10 @@
#pragma once
+#include <memory>
+#include <string>
+#include <unordered_map>
+
#include <arrow/c/abi.h>
#include <arrow/memory_pool.h>
#include <arrow/record_batch.h>
@@ -27,12 +31,12 @@
namespace gluten {
-class VeloxDataSource {
+class GlutenDataSource {
public:
- VeloxDataSource(const std::string& filePath, std::shared_ptr<arrow::Schema> schema)
+ GlutenDataSource(const std::string& filePath, std::shared_ptr<arrow::Schema> schema)
: filePath_(filePath), schema_(schema) {}
- virtual ~VeloxDataSource() = default;
+ virtual ~GlutenDataSource() = default;
virtual void init(const std::unordered_map<std::string, std::string>& sparkConfs) {}
virtual void inspectSchema(struct ArrowSchema* out) = 0;
diff --git a/cpp/velox/tests/FilePathGenerator.h b/cpp/core/tests/FilePathGenerator.h
similarity index 100%
copy from cpp/velox/tests/FilePathGenerator.h
copy to cpp/core/tests/FilePathGenerator.h
diff --git a/cpp/velox/tests/JsonToProtoConverter.cc b/cpp/core/tests/JsonToProtoConverter.cc
similarity index 75%
copy from cpp/velox/tests/JsonToProtoConverter.cc
copy to cpp/core/tests/JsonToProtoConverter.cc
index 4ed1372d1f..a5498f755e 100644
--- a/cpp/velox/tests/JsonToProtoConverter.cc
+++ b/cpp/core/tests/JsonToProtoConverter.cc
@@ -15,22 +15,24 @@
* limitations under the License.
*/
-#include "utils//JsonToProtoConverter.h"
+#include "../utils/JsonToProtoConverter.h"
+
+#include <cerrno>
+#include <cstring>
#include <fstream>
#include <sstream>
-#include "velox/common/base/Exceptions.h"
+
+#include "../utils/Exception.h"
void JsonToProtoConverter::readFromFile(const std::string& msgPath, google::protobuf::Message& msg) {
- // Read json file and resume the Substrait plan.
std::ifstream msgJson(msgPath);
- VELOX_CHECK(!msgJson.fail(), "Failed to open file: {}. {}", msgPath, strerror(errno));
+ GLUTEN_CHECK(!msgJson.fail(), std::string("Failed to open file: ") + msgPath + ". " + strerror(errno));
std::stringstream buffer;
buffer << msgJson.rdbuf();
std::string msgData = buffer.str();
auto status = google::protobuf::util::JsonStringToMessage(msgData, &msg);
- VELOX_CHECK(
+ GLUTEN_CHECK(
status.ok(),
- "Failed to parse Substrait JSON: {} {}",
- static_cast<int8_t>(status.code()),
- status.message().ToString());
+ "Failed to parse Substrait JSON: " + std::to_string(static_cast<int>(status.code())) + " " +
+ status.message().ToString());
}
diff --git a/cpp/velox/utils/JsonToProtoConverter.h b/cpp/core/tests/JsonToProtoConverter.h
similarity index 98%
copy from cpp/velox/utils/JsonToProtoConverter.h
copy to cpp/core/tests/JsonToProtoConverter.h
index f9c41bb695..0ef9cd4dd8 100644
--- a/cpp/velox/utils/JsonToProtoConverter.h
+++ b/cpp/core/tests/JsonToProtoConverter.h
@@ -17,6 +17,8 @@
#pragma once
+#include <string>
+
#include <google/protobuf/util/json_util.h>
class JsonToProtoConverter {
diff --git a/cpp/velox/tests/utils/TestAllocationListener.h b/cpp/core/tests/utils/TestAllocationListener.h
similarity index 96%
copy from cpp/velox/tests/utils/TestAllocationListener.h
copy to cpp/core/tests/utils/TestAllocationListener.h
index 9a94709aa3..3e51bf2aa4 100644
--- a/cpp/velox/tests/utils/TestAllocationListener.h
+++ b/cpp/core/tests/utils/TestAllocationListener.h
@@ -17,13 +17,14 @@
#pragma once
+#include <limits>
+
#include "compute/ResultIterator.h"
#include "memory/AllocationListener.h"
#include "shuffle/ShuffleWriter.h"
namespace gluten {
-// instance with limited capacity, used by tests and benchmarks.
class TestAllocationListener final : public AllocationListener {
public:
TestAllocationListener() = default;
diff --git a/cpp/velox/tests/utils/TestStreamReader.h b/cpp/core/tests/utils/TestStreamReader.h
similarity index 97%
copy from cpp/velox/tests/utils/TestStreamReader.h
copy to cpp/core/tests/utils/TestStreamReader.h
index fd47574c6b..184d3ab671 100644
--- a/cpp/velox/tests/utils/TestStreamReader.h
+++ b/cpp/core/tests/utils/TestStreamReader.h
@@ -17,6 +17,9 @@
#pragma once
+#include <memory>
+#include <utility>
+
#include "shuffle/ShuffleReader.h"
#include "shuffle/ShuffleWriter.h"
diff --git a/cpp/velox/tests/utils/TestUtils.h b/cpp/core/tests/utils/TestUtils.h
similarity index 98%
copy from cpp/velox/tests/utils/TestUtils.h
copy to cpp/core/tests/utils/TestUtils.h
index 5de4b729f3..880e5eef71 100644
--- a/cpp/velox/tests/utils/TestUtils.h
+++ b/cpp/core/tests/utils/TestUtils.h
@@ -17,6 +17,9 @@
#pragma once
+#include <stdexcept>
+
+
#define ASSERT_NOT_OK(status) \
do { \
arrow::Status __s = (status); \
diff --git a/cpp/velox/udf/Udaf.h b/cpp/core/udf/Udaf.h
similarity index 100%
copy from cpp/velox/udf/Udaf.h
copy to cpp/core/udf/Udaf.h
diff --git a/cpp/velox/udf/Udf.h b/cpp/core/udf/Udf.h
similarity index 100%
copy from cpp/velox/udf/Udf.h
copy to cpp/core/udf/Udf.h
diff --git a/cpp/velox/utils/JsonToProtoConverter.h b/cpp/core/utils/JsonToProtoConverter.h
similarity index 98%
copy from cpp/velox/utils/JsonToProtoConverter.h
copy to cpp/core/utils/JsonToProtoConverter.h
index f9c41bb695..0ef9cd4dd8 100644
--- a/cpp/velox/utils/JsonToProtoConverter.h
+++ b/cpp/core/utils/JsonToProtoConverter.h
@@ -17,6 +17,8 @@
#pragma once
+#include <string>
+
#include <google/protobuf/util/json_util.h>
class JsonToProtoConverter {
diff --git a/cpp/velox/cudf/CudfPlanValidator.h b/cpp/velox/cudf/CudfPlanValidator.h
index de4fb150e0..7b08d1d57d 100644
--- a/cpp/velox/cudf/CudfPlanValidator.h
+++ b/cpp/velox/cudf/CudfPlanValidator.h
@@ -17,13 +17,4 @@
#pragma once
-#include "memory/VeloxMemoryManager.h"
-#include "substrait/plan.pb.h"
-
-namespace gluten {
-class CudfPlanValidator {
- public:
- // Validate if the plan contains cudf unsupported operator except TableScan and ValueStream.
- static bool validate(const ::substrait::Plan& substraitPlan);
-};
-} // namespace gluten
+#include "../../core/cudf/CudfPlanValidator.h"
diff --git a/cpp/velox/jni/JniUdf.cc b/cpp/velox/jni/JniUdf.cc
index 32514e7ae7..5b72be3e09 100644
--- a/cpp/velox/jni/JniUdf.cc
+++ b/cpp/velox/jni/JniUdf.cc
@@ -16,79 +16,15 @@
*/
#include "JniUdf.h"
-#include "jni/JniCommon.h"
-#include "udf/UdfLoader.h"
-#include "utils/Exception.h"
-namespace {
+namespace gluten {
-static JavaVM* vm;
-
-const std::string kUdfResolverClassPath = "Lorg/apache/spark/sql/expression/UDFResolver$;";
-
-static jclass udfResolverClass;
-static jmethodID registerUDFMethod;
-static jmethodID registerUDAFMethod;
-
-} // namespace
-
-void gluten::initVeloxJniUDF(JNIEnv* env) {
- if (env->GetJavaVM(&vm) != JNI_OK) {
- throw gluten::GlutenException("Unable to get JavaVM instance");
- }
-
- // classes
- udfResolverClass = createGlobalClassReferenceOrError(env, kUdfResolverClassPath.c_str());
-
- // methods
- registerUDFMethod = getMethodIdOrError(env, udfResolverClass, "registerUDF", "(Ljava/lang/String;[B[BZZ)V");
- registerUDAFMethod = getMethodIdOrError(env, udfResolverClass, "registerUDAF", "(Ljava/lang/String;[B[B[BZZ)V");
+void initVeloxJniUDF(JNIEnv* env) {
+ initJniUDF(env);
}
-void gluten::finalizeVeloxJniUDF(JNIEnv* env) {
- env->DeleteGlobalRef(udfResolverClass);
+void finalizeVeloxJniUDF(JNIEnv* env) {
+ finalizeJniUDF(env);
}
-void gluten::jniRegisterFunctionSignatures(JNIEnv* env) {
- auto udfLoader = gluten::UdfLoader::getInstance();
-
- const auto& signatures = udfLoader->getRegisteredUdfSignatures();
- for (const auto& signature : signatures) {
- jstring name = env->NewStringUTF(signature->name.c_str());
- jbyteArray returnType = env->NewByteArray(signature->returnType.length());
- env->SetByteArrayRegion(
- returnType, 0, signature->returnType.length(), reinterpret_cast<const jbyte*>(signature->returnType.c_str()));
- jbyteArray argTypes = env->NewByteArray(signature->argTypes.length());
- env->SetByteArrayRegion(
- argTypes, 0, signature->argTypes.length(), reinterpret_cast<const jbyte*>(signature->argTypes.c_str()));
- jobject instance = env->GetStaticObjectField(
- udfResolverClass, env->GetStaticFieldID(udfResolverClass, "MODULE$", kUdfResolverClassPath.c_str()));
- if (!signature->intermediateType.empty()) {
- jbyteArray intermediateType = env->NewByteArray(signature->intermediateType.length());
- env->SetByteArrayRegion(
- intermediateType,
- 0,
- signature->intermediateType.length(),
- reinterpret_cast<const jbyte*>(signature->intermediateType.c_str()));
- env->CallVoidMethod(
- instance,
- registerUDAFMethod,
- name,
- returnType,
- argTypes,
- intermediateType,
- signature->variableArity,
- signature->allowTypeConversion);
- } else {
- env->CallVoidMethod(
- instance,
- registerUDFMethod,
- name,
- returnType,
- argTypes,
- signature->variableArity,
- signature->allowTypeConversion);
- }
- checkException(env);
- }
-}
+} // namespace gluten
diff --git a/cpp/velox/jni/JniUdf.h b/cpp/velox/jni/JniUdf.h
index 568439d184..5c4ba2c996 100644
--- a/cpp/velox/jni/JniUdf.h
+++ b/cpp/velox/jni/JniUdf.h
@@ -18,8 +18,8 @@
#pragma once
#include <jni.h>
-#include <string>
-#include <unordered_map>
+
+#include "../../core/jni/JniUdf.h"
namespace gluten {
@@ -27,6 +27,4 @@ void initVeloxJniUDF(JNIEnv* env);
void finalizeVeloxJniUDF(JNIEnv* env);
-void jniRegisterFunctionSignatures(JNIEnv* env);
-
} // namespace gluten
diff --git a/cpp/velox/operators/functions/Arithmetic.h b/cpp/velox/operators/functions/Arithmetic.h
index 9b2e22ac41..045bda7bcd 100644
--- a/cpp/velox/operators/functions/Arithmetic.h
+++ b/cpp/velox/operators/functions/Arithmetic.h
@@ -14,53 +14,4 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include <folly/CPortability.h>
-#include <stdint.h>
-#include <cmath>
-#include <limits>
-#include <type_traits>
-
-namespace gluten {
-template <typename T>
-/// Round function
-/// When AlwaysRoundNegDec is true, spark semantics is followed which
-/// rounds negative decimals for integrals and does not round it otherwise
-/// Note that is is likely techinically impossible for this function to return
-/// expected results in all cases as the loss of precision plagues it on both
-/// paths: factor multiplication for large numbers and addition of truncated
-/// number to the rounded fraction for small numbers.
-/// We are trying to minimize the loss of precision by using the best path for
-/// the number, but the journey is likely not over yet.
-struct RoundFunction {
- template <typename TNum, typename TDecimals, bool alwaysRoundNegDec = true>
- FOLLY_ALWAYS_INLINE TNum round(const TNum& number, const TDecimals& decimals = 0) {
- static_assert(!std::is_same_v<TNum, bool> && "round not supported for bool");
-
- if constexpr (std::is_integral_v<TNum>) {
- if constexpr (alwaysRoundNegDec) {
- if (decimals >= 0)
- return number;
- } else {
- return number;
- }
- }
- if (!std::isfinite(number)) {
- return number;
- }
-
- // Using long double for high precision during intermediate calculations.
- // TODO: Make this more efficient with Boost to support high arbitrary precision at runtime.
- long double factor = std::pow(10.0L, static_cast<long double>(decimals));
- static const TNum kInf = std::numeric_limits<TNum>::infinity();
-
- if (number < 0) {
- return static_cast<TNum>((std::round(std::nextafter(number, -kInf) * factor * -1) / factor) * -1);
- }
- return static_cast<TNum>(std::round(std::nextafter(number, kInf) * factor) / factor);
- }
- template <typename TInput>
- FOLLY_ALWAYS_INLINE void call(TInput& result, const TInput& a, const int32_t b = 0) {
- result = round(a, b);
- }
-};
-} // namespace gluten
+#include "../../../core/operators/functions/Arithmetic.h"
diff --git a/cpp/velox/operators/functions/RegistrationAllFunctions.h b/cpp/velox/operators/functions/RegistrationAllFunctions.h
index 9ba754d14f..319579b1ec 100644
--- a/cpp/velox/operators/functions/RegistrationAllFunctions.h
+++ b/cpp/velox/operators/functions/RegistrationAllFunctions.h
@@ -17,8 +17,4 @@
#pragma once
-namespace gluten {
-
-void registerAllFunctions();
-
-} // namespace gluten
+#include "../../../core/operators/functions/RegistrationAllFunctions.h"
diff --git a/cpp/velox/operators/writer/VeloxDataSource.h b/cpp/velox/operators/writer/VeloxDataSource.h
index 043f1b8b17..7c4b8ae29f 100644
--- a/cpp/velox/operators/writer/VeloxDataSource.h
+++ b/cpp/velox/operators/writer/VeloxDataSource.h
@@ -17,32 +17,10 @@
#pragma once
-#include <arrow/c/abi.h>
-#include <arrow/memory_pool.h>
-#include <arrow/record_batch.h>
-#include <arrow/type.h>
-#include <arrow/type_fwd.h>
-
-#include "memory/ColumnarBatch.h"
+#include "../../../core/operators/writer/GlutenDataSource.h"
namespace gluten {
-class VeloxDataSource {
- public:
- VeloxDataSource(const std::string& filePath, std::shared_ptr<arrow::Schema> schema)
- : filePath_(filePath), schema_(schema) {}
-
- virtual ~VeloxDataSource() = default;
-
- virtual void init(const std::unordered_map<std::string, std::string>& sparkConfs) {}
- virtual void inspectSchema(struct ArrowSchema* out) = 0;
- virtual void write(const std::shared_ptr<ColumnarBatch>& cb) {}
- virtual void close() {}
- virtual std::shared_ptr<arrow::Schema> getSchema() = 0;
-
- private:
- std::string filePath_;
- std::shared_ptr<arrow::Schema> schema_;
-};
+using VeloxDataSource = GlutenDataSource;
} // namespace gluten
diff --git a/cpp/velox/tests/FilePathGenerator.h b/cpp/velox/tests/FilePathGenerator.h
index 0858fcc2e0..331043dc90 100644
--- a/cpp/velox/tests/FilePathGenerator.h
+++ b/cpp/velox/tests/FilePathGenerator.h
@@ -17,9 +17,4 @@
#pragma once
-#include <string>
-
-class FilePathGenerator {
- public:
- static const std::string getDataFilePath(const std::string& fileName);
-};
+#include "../../core/tests/FilePathGenerator.h"
diff --git a/cpp/velox/tests/JsonToProtoConverter.cc b/cpp/velox/tests/JsonToProtoConverter.cc
index 4ed1372d1f..1c422d5740 100644
--- a/cpp/velox/tests/JsonToProtoConverter.cc
+++ b/cpp/velox/tests/JsonToProtoConverter.cc
@@ -15,22 +15,4 @@
* limitations under the License.
*/
-#include "utils//JsonToProtoConverter.h"
-#include <fstream>
-#include <sstream>
-#include "velox/common/base/Exceptions.h"
-
-void JsonToProtoConverter::readFromFile(const std::string& msgPath, google::protobuf::Message& msg) {
- // Read json file and resume the Substrait plan.
- std::ifstream msgJson(msgPath);
- VELOX_CHECK(!msgJson.fail(), "Failed to open file: {}. {}", msgPath, strerror(errno));
- std::stringstream buffer;
- buffer << msgJson.rdbuf();
- std::string msgData = buffer.str();
- auto status = google::protobuf::util::JsonStringToMessage(msgData, &msg);
- VELOX_CHECK(
- status.ok(),
- "Failed to parse Substrait JSON: {} {}",
- static_cast<int8_t>(status.code()),
- status.message().ToString());
-}
+#include "../../core/tests/JsonToProtoConverter.cc"
diff --git a/cpp/velox/tests/JsonToProtoConverter.h b/cpp/velox/tests/JsonToProtoConverter.h
index f9c41bb695..f594ccc0a0 100644
--- a/cpp/velox/tests/JsonToProtoConverter.h
+++ b/cpp/velox/tests/JsonToProtoConverter.h
@@ -17,10 +17,4 @@
#pragma once
-#include <google/protobuf/util/json_util.h>
-
-class JsonToProtoConverter {
- public:
- /// Reconstruct Protobuf message from Json file.
- static void readFromFile(const std::string& msgPath, google::protobuf::Message& msg);
-};
+#include "../../core/tests/JsonToProtoConverter.h"
diff --git a/cpp/velox/tests/utils/TestAllocationListener.h b/cpp/velox/tests/utils/TestAllocationListener.h
index 9a94709aa3..980c41ef53 100644
--- a/cpp/velox/tests/utils/TestAllocationListener.h
+++ b/cpp/velox/tests/utils/TestAllocationListener.h
@@ -17,50 +17,4 @@
#pragma once
-#include "compute/ResultIterator.h"
-#include "memory/AllocationListener.h"
-#include "shuffle/ShuffleWriter.h"
-
-namespace gluten {
-
-// instance with limited capacity, used by tests and benchmarks.
-class TestAllocationListener final : public AllocationListener {
- public:
- TestAllocationListener() = default;
-
- void setThrowIfOOM(bool throwIfOOM) {
- throwIfOOM_ = throwIfOOM;
- }
-
- void updateLimit(uint64_t limit) {
- limit_ = limit;
- }
-
- void setIterator(ResultIterator* iterator) {
- iterator_ = iterator;
- }
-
- void setShuffleWriter(ShuffleWriter* shuffleWriter) {
- shuffleWriter_ = shuffleWriter;
- }
-
- void allocationChanged(int64_t diff) override;
-
- int64_t currentBytes() override;
-
- int64_t reclaimedBytes() const;
-
- void reset();
-
- private:
- bool throwIfOOM_{false};
-
- uint64_t usedBytes_{0L};
- uint64_t reclaimedBytes_{0L};
-
- uint64_t limit_{std::numeric_limits<uint64_t>::max()};
- ResultIterator* iterator_{nullptr};
- ShuffleWriter* shuffleWriter_{nullptr};
-};
-
-} // namespace gluten
+#include "../../../core/tests/utils/TestAllocationListener.h"
diff --git a/cpp/velox/tests/utils/TestStreamReader.h b/cpp/velox/tests/utils/TestStreamReader.h
index fd47574c6b..0c14395b15 100644
--- a/cpp/velox/tests/utils/TestStreamReader.h
+++ b/cpp/velox/tests/utils/TestStreamReader.h
@@ -17,21 +17,4 @@
#pragma once
-#include "shuffle/ShuffleReader.h"
-#include "shuffle/ShuffleWriter.h"
-
-namespace gluten {
-
-class TestStreamReader : public StreamReader {
- public:
- explicit TestStreamReader(const std::shared_ptr<arrow::io::InputStream>& inputStream) : inputStream_(inputStream) {}
-
- std::shared_ptr<arrow::io::InputStream> readNextStream(arrow::MemoryPool*) override {
- return std::move(inputStream_);
- }
-
- private:
- std::shared_ptr<arrow::io::InputStream> inputStream_;
-};
-
-} // namespace gluten
+#include "../../../core/tests/utils/TestStreamReader.h"
diff --git a/cpp/velox/tests/utils/TestUtils.h b/cpp/velox/tests/utils/TestUtils.h
index 5de4b729f3..817ba2b552 100644
--- a/cpp/velox/tests/utils/TestUtils.h
+++ b/cpp/velox/tests/utils/TestUtils.h
@@ -17,10 +17,4 @@
#pragma once
-#define ASSERT_NOT_OK(status) \
- do { \
- arrow::Status __s = (status); \
- if (!__s.ok()) { \
- throw std::runtime_error(__s.message()); \
- } \
- } while (false);
+#include "../../../core/tests/utils/TestUtils.h"
diff --git a/cpp/velox/udf/Udaf.h b/cpp/velox/udf/Udaf.h
index f421ecbc31..31f364da79 100644
--- a/cpp/velox/udf/Udaf.h
+++ b/cpp/velox/udf/Udaf.h
@@ -17,27 +17,4 @@
#pragma once
-namespace gluten {
-
-struct UdafEntry {
- const char* name;
- const char* dataType;
-
- int numArgs;
- const char** argTypes;
-
- const char* intermediateType{nullptr};
- bool variableArity{false};
- bool allowTypeConversion{false};
-};
-
-#define GLUTEN_GET_NUM_UDAF getNumUdaf
-#define DEFINE_GET_NUM_UDAF extern "C" int GLUTEN_GET_NUM_UDAF()
-
-#define GLUTEN_GET_UDAF_ENTRIES getUdfEntries
-#define DEFINE_GET_UDAF_ENTRIES extern "C" void GLUTEN_GET_UDAF_ENTRIES(gluten::UdafEntry* udafEntries)
-
-#define GLUTEN_REGISTER_UDAF registerUdf
-#define DEFINE_REGISTER_UDAF extern "C" void GLUTEN_REGISTER_UDAF()
-
-} // namespace gluten
+#include "../../core/udf/Udaf.h"
diff --git a/cpp/velox/udf/Udf.h b/cpp/velox/udf/Udf.h
index e0b3a70004..a31b50f182 100644
--- a/cpp/velox/udf/Udf.h
+++ b/cpp/velox/udf/Udf.h
@@ -17,26 +17,4 @@
#pragma once
-namespace gluten {
-
-struct UdfEntry {
- const char* name;
- const char* dataType;
-
- int numArgs;
- const char** argTypes;
-
- bool variableArity{false};
- bool allowTypeConversion{false};
-};
-
-#define GLUTEN_GET_NUM_UDF getNumUdf
-#define DEFINE_GET_NUM_UDF extern "C" int GLUTEN_GET_NUM_UDF()
-
-#define GLUTEN_GET_UDF_ENTRIES getUdfEntries
-#define DEFINE_GET_UDF_ENTRIES extern "C" void GLUTEN_GET_UDF_ENTRIES(gluten::UdfEntry* udfEntries)
-
-#define GLUTEN_REGISTER_UDF registerUdf
-#define DEFINE_REGISTER_UDF extern "C" void GLUTEN_REGISTER_UDF()
-
-} // namespace gluten
+#include "../../core/udf/Udf.h"
diff --git a/cpp/velox/utils/JsonToProtoConverter.h b/cpp/velox/utils/JsonToProtoConverter.h
index f9c41bb695..137206312f 100644
--- a/cpp/velox/utils/JsonToProtoConverter.h
+++ b/cpp/velox/utils/JsonToProtoConverter.h
@@ -17,10 +17,4 @@
#pragma once
-#include <google/protobuf/util/json_util.h>
-
-class JsonToProtoConverter {
- public:
- /// Reconstruct Protobuf message from Json file.
- static void readFromFile(const std::string& msgPath, google::protobuf::Message& msg);
-};
+#include "../../core/utils/JsonToProtoConverter.h"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 13/38: [GLUTEN][CORE] Add common Paimon scan hooks and metadata plumbing
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 33107d98a199c5932c6b2a20295ffde959c9756d
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 19:05:15 2026 +0800
[GLUTEN][CORE] Add common Paimon scan hooks and metadata plumbing
Move Paimon scan validation and filter policy behind backend hooks while sharing the split metadata and scan transformer plumbing needed by native backends.
---
.../substrait/rel/PaimonLocalFilesBuilder.java | 18 +-
.../gluten/substrait/rel/PaimonLocalFilesNode.java | 54 +++-
.../gluten/execution/OffloadPaimonScan.scala | 4 +-
.../gluten/execution/PaimonScanTransformer.scala | 303 ++++++++++++++-------
...floadPaimonScan.scala => PaimonSparkShim.scala} | 23 +-
.../gluten/execution/PaimonSparkShimImpl.scala | 46 ++++
.../apache/paimon/spark/source/PaimonConfig.scala | 57 ++++
.../substrait/proto/substrait/algebra.proto | 15 +
.../gluten/backendsapi/SparkPlanExecApi.scala | 21 ++
.../execution/BasicScanExecTransformer.scala | 5 +-
10 files changed, 438 insertions(+), 108 deletions(-)
diff --git a/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesBuilder.java b/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesBuilder.java
index 4d6c33baee..d7b69eebc6 100644
--- a/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesBuilder.java
+++ b/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesBuilder.java
@@ -28,7 +28,14 @@ public class PaimonLocalFilesBuilder {
List<Map<String, String>> partitionColumns,
LocalFilesNode.ReadFileFormat fileFormat,
List<String> preferredLocations,
- Map<String, String> properties) {
+ Map<String, String> properties,
+ List<Integer> buckets,
+ List<Long> firstRowIds,
+ List<Long> maxSequenceNumbers,
+ List<Integer> splitGroups,
+ boolean useHiveSplit,
+ List<String> primaryKeys,
+ boolean allRawConvertible) {
return new PaimonLocalFilesNode(
index,
paths,
@@ -37,6 +44,13 @@ public class PaimonLocalFilesBuilder {
partitionColumns,
fileFormat,
preferredLocations,
- properties);
+ properties,
+ buckets,
+ firstRowIds,
+ maxSequenceNumbers,
+ splitGroups,
+ useHiveSplit,
+ primaryKeys,
+ allRawConvertible);
}
}
diff --git a/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesNode.java b/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesNode.java
index 30f47d87b3..d78a6c224d 100644
--- a/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesNode.java
+++ b/gluten-paimon/src-paimon/main/java/org/apache/gluten/substrait/rel/PaimonLocalFilesNode.java
@@ -16,11 +16,23 @@
*/
package org.apache.gluten.substrait.rel;
+import io.substrait.proto.ReadRel;
+import io.substrait.proto.ReadRel.LocalFiles.FileOrFiles.OrcReadOptions;
+import io.substrait.proto.ReadRel.LocalFiles.FileOrFiles.PaimonReadOptions;
+import io.substrait.proto.ReadRel.LocalFiles.FileOrFiles.ParquetReadOptions;
+
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
public class PaimonLocalFilesNode extends LocalFilesNode {
+ private final List<Integer> buckets;
+ private final List<Long> firstRowIds;
+ private final List<Long> maxSequenceNumbers;
+ private final List<Integer> splitGroups;
+ private final boolean useHiveSplit;
+ private final List<String> primaryKeys;
+ private final boolean allRawConvertible;
public PaimonLocalFilesNode(
Integer index,
@@ -30,7 +42,14 @@ public class PaimonLocalFilesNode extends LocalFilesNode {
List<Map<String, String>> partitionColumns,
ReadFileFormat fileFormat,
List<String> preferredLocations,
- Map<String, String> properties) {
+ Map<String, String> properties,
+ List<Integer> buckets,
+ List<Long> firstRowIds,
+ List<Long> maxSequenceNumbers,
+ List<Integer> splitGroups,
+ boolean useHiveSplit,
+ List<String> primaryKeys,
+ boolean allRawConvertible) {
super(
index,
paths,
@@ -44,5 +63,38 @@ public class PaimonLocalFilesNode extends LocalFilesNode {
preferredLocations,
properties,
new ArrayList<>());
+ this.buckets = buckets;
+ this.firstRowIds = firstRowIds;
+ this.maxSequenceNumbers = maxSequenceNumbers;
+ this.splitGroups = splitGroups;
+ this.useHiveSplit = useHiveSplit;
+ this.primaryKeys = primaryKeys;
+ this.allRawConvertible = allRawConvertible;
+ }
+
+ @Override
+ protected void processFileBuilder(ReadRel.LocalFiles.FileOrFiles.Builder fileBuilder, int index) {
+ PaimonReadOptions.Builder paimonBuilder = PaimonReadOptions.newBuilder();
+ paimonBuilder.setBucket(buckets.get(index));
+ paimonBuilder.setFirstRowId(firstRowIds.get(index));
+ paimonBuilder.setMaxSequenceNumber(maxSequenceNumbers.get(index));
+ paimonBuilder.setSplitGroup(splitGroups.get(index));
+ paimonBuilder.setUseHiveSplit(useHiveSplit);
+ paimonBuilder.addAllPrimaryKeys(primaryKeys);
+ paimonBuilder.setRawConvertible(allRawConvertible);
+
+ switch (fileFormat) {
+ case ParquetReadFormat:
+ paimonBuilder.setParquet(ParquetReadOptions.newBuilder().build());
+ break;
+ case OrcReadFormat:
+ paimonBuilder.setOrc(OrcReadOptions.newBuilder().build());
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported file format " + fileFormat.name() + " for paimon data file.");
+ }
+
+ fileBuilder.setPaimon(paimonBuilder);
}
}
diff --git a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala
index 19c0c22433..18f2151e6e 100644
--- a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala
+++ b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala
@@ -23,8 +23,8 @@ import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
case class OffloadPaimonScan() extends OffloadSingleNode {
override def offload(plan: SparkPlan): SparkPlan = plan match {
- case scan: BatchScanExec if PaimonScanTransformer.supportsBatchScan(scan.scan) =>
- PaimonScanTransformer(scan)
+ case scan: BatchScanExec if AbstractPaimonScanTransformer.supportsBatchScan(scan.scan).ok() =>
+ AbstractPaimonScanTransformer(scan)
case other => other
}
}
diff --git a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonScanTransformer.scala b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonScanTransformer.scala
index 12bccb1568..927773af78 100644
--- a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonScanTransformer.scala
+++ b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonScanTransformer.scala
@@ -16,37 +16,44 @@
*/
package org.apache.gluten.execution
+import org.apache.gluten.backendsapi.BackendsApiManager
import org.apache.gluten.exception.GlutenNotSupportException
import org.apache.gluten.sql.shims.SparkShimLoader
import org.apache.gluten.substrait.rel.{PaimonLocalFilesBuilder, SplitInfo}
import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat
import org.apache.spark.Partition
+import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.softaffinity.SoftAffinity
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, DynamicPruningExpression, Expression, Literal}
import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.connector.catalog.Table
import org.apache.spark.sql.connector.read.Scan
import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.vectorized.ColumnarBatch
import org.apache.paimon.CoreOptions
-import org.apache.paimon.spark.{PaimonInputPartition, PaimonScan}
+import org.apache.paimon.spark.{PaimonBaseScan, PaimonInputPartition, PaimonScan}
+import org.apache.paimon.spark.schema.PaimonMetadataColumn.SUPPORTED_METADATA_COLUMNS
+import org.apache.paimon.spark.source.PaimonConfig
import org.apache.paimon.table.{DataTable, FileStoreTable}
import org.apache.paimon.table.source.DataSplit
-import org.apache.paimon.utils.InternalRowPartitionComputer
+import org.apache.paimon.types.DecimalType
+import java.lang.{Integer => JInteger}
import java.lang.{Long => JLong}
import java.util.{HashMap => JHashMap, Map => JMap}
import scala.collection.JavaConverters._
+import scala.collection.immutable.HashMap
import scala.collection.mutable
-case class PaimonScanTransformer(
+abstract class AbstractPaimonScanTransformer(
override val output: Seq[AttributeReference],
@transient override val scan: Scan,
override val runtimeFilters: Seq[Expression],
@@ -58,14 +65,36 @@ case class PaimonScanTransformer(
output = output,
scan = scan,
runtimeFilters = runtimeFilters,
- table = table,
keyGroupedPartitioning = keyGroupedPartitioning,
+ table = table,
commonPartitionValues = commonPartitionValues
- ) {
+ )
+ with Logging {
+
+ @transient private lazy val shim: PaimonSparkShim = new PaimonSparkShimImpl()
+
+ protected lazy val tableProperties: HashMap[String, String] = {
+ val map = HashMap.newBuilder[String, String]
+ scan match {
+ case paimonScan: PaimonScan =>
+ val paimonTable = paimonScan.table
+ val coreOptions = new CoreOptions(paimonTable.options())
+ coreOptions.toMap.forEach((key, value) => map += (key -> value))
+ if (SQLConf.get.getConf(PaimonConfig.PAIMON_NATIVE_SPLIT_ENABLED)) {
+ map += ("isPaimon" -> "true")
+ }
+ val primaryKeys = paimonTable.primaryKeys()
+ if (!primaryKeys.isEmpty) {
+ map += ("primary-key" -> String.join(",", primaryKeys))
+ }
+ case _ =>
+ }
+ map.result()
+ }
private lazy val coreOptions: CoreOptions = scan match {
- case scan: PaimonScan =>
- scan.table match {
+ case paimonScan: PaimonScan =>
+ paimonScan.table match {
case dataTable: DataTable =>
dataTable.coreOptions()
case _ =>
@@ -83,15 +112,8 @@ case class PaimonScanTransformer(
throw new GlutenNotSupportException("Only support PaimonScan.")
}
- override def getDataSchema: StructType = new StructType()
-
- override def withNewPushdownFilters(filters: Seq[Expression]): PaimonScanTransformer = {
- this.copy(pushDownFilters = Some(filters))
- }
-
- override def withOutput(newOutput: Seq[AttributeReference]): BatchScanExecTransformerBase = {
- this.copy(output = newOutput)
- }
+ override def getDataSchema: StructType = CharVarcharUtils.replaceCharVarcharWithStringInSchema(
+ scan.asInstanceOf[PaimonBaseScan].readSchema())
override lazy val fileFormat: ReadFileFormat = {
val formatStr = coreOptions.fileFormatString()
@@ -105,100 +127,113 @@ case class PaimonScanTransformer(
}
override def doValidateInternal(): ValidationResult = {
- scan match {
- case paimonScan: PaimonScan =>
- paimonScan.table match {
- case table: FileStoreTable =>
- if (fileFormat == ReadFileFormat.UnknownFormat) {
- return ValidationResult.failed("Only support parquet/orc Paimon table.")
- }
- if (!table.primaryKeys().isEmpty || coreOptions.deletionVectorsEnabled()) {
- return ValidationResult.failed("Not support Paimon PK/DV table.")
- }
- case table =>
- return ValidationResult.failed(
- s"Not support Paimon ${table.getClass.getSimpleName} table.")
- }
- case _ =>
- return ValidationResult.failed("Only support PaimonScan.")
+ val result = AbstractPaimonScanTransformer.supportsBatchScan(scan)
+ if (result.ok()) {
+ return super.doValidateInternal()
}
- super.doValidateInternal()
+ result
}
override def doExecuteColumnar(): RDD[ColumnarBatch] = throw new UnsupportedOperationException()
+ override protected def rewriteNativeScanFilters(filters: Seq[Expression]): Seq[Expression] =
+ scan match {
+ case paimonScan: PaimonScan =>
+ BackendsApiManager.getSparkPlanExecApiInstance.rewritePaimonPushdownFilters(
+ filters,
+ paimonScan.table.primaryKeys().asScala.toSet,
+ SUPPORTED_METADATA_COLUMNS.toSet)
+ case _ => filters
+ }
+
override def getSplitInfosFromPartitions(
partitions: Seq[(Partition, ReadFileFormat)]): Seq[SplitInfo] = {
- val partitionComputer = PaimonScanTransformer.getRowDataPartitionComputer(scan)
- partitions.map { case (partition, _) => partitionToSplitInfo(partition, partitionComputer) }
- }
+ val partitionComputer = scan match {
+ case paimonScan: PaimonScan => shim.getInternalPartitionComputer(paimonScan)
+ case _ => throw new GlutenNotSupportException("Only support PaimonScan.")
+ }
+ val paimonScan = scan.asInstanceOf[PaimonScan]
+ val primaryKeys = paimonScan.table.primaryKeys()
+ val useHiveSplit = !SQLConf.get.getConf(PaimonConfig.PAIMON_NATIVE_SPLIT_ENABLED)
- private def partitionToSplitInfo(
- partition: Partition,
- partitionComputer: InternalRowPartitionComputer): SplitInfo = {
- partition match {
- case p: SparkDataSourceRDDPartition =>
+ partitions.map {
+ case (partition: SparkDataSourceRDDPartition, _) =>
val paths = mutable.ListBuffer.empty[String]
val starts = mutable.ListBuffer.empty[JLong]
val lengths = mutable.ListBuffer.empty[JLong]
val partitionColumns = mutable.ListBuffer.empty[JMap[String, String]]
+ val buckets = mutable.ListBuffer.empty[JInteger]
+ val firstRowIds = mutable.ListBuffer.empty[JLong]
+ val maxSequenceNumbers = mutable.ListBuffer.empty[JLong]
+ val splitGroups = mutable.ListBuffer.empty[JInteger]
+ val allRawConvertible = partition.inputPartitions.forall {
+ case paimonPartition: PaimonInputPartition =>
+ paimonPartition.splits.forall(_.convertToRawFiles().isPresent)
+ case _ => false
+ }
- p.inputPartitions.foreach {
- case partition: PaimonInputPartition =>
- partition.splits.foreach {
- split =>
- val rawFilesOpt = split.convertToRawFiles()
- if (rawFilesOpt.isPresent) {
- val partitionRow = split.asInstanceOf[DataSplit].partition()
- val partitionCols = partitionComputer.generatePartValues(partitionRow)
- val rawFiles = rawFilesOpt.get().asScala
- paths ++= rawFiles.map(_.path())
- starts ++= rawFiles.map(file => JLong.valueOf(file.offset()))
- lengths ++= rawFiles.map(file => JLong.valueOf(file.length()))
- partitionColumns ++= mutable.ArrayBuffer.fill(rawFiles.size)(partitionCols)
- } else {
- throw new GlutenNotSupportException(
- "Cannot get raw files from paimon SparkInputPartition.")
- }
+ partition.inputPartitions.foreach {
+ case paimonPartition: PaimonInputPartition =>
+ paimonPartition.splits.zipWithIndex.foreach {
+ case (split: DataSplit, splitIdx) =>
+ val partitionRow =
+ partitionComputer.generatePartValues(shim.getSplitPartition(split))
+ val fileMetas = split.dataFiles().asScala
+ val bucket = split.bucket()
+ paths ++= fileMetas.map(
+ file => {
+ val bucketPath = shim.getBucketPath(split, file)
+ bucketPath + "/" + file.fileName()
+ })
+ starts ++= mutable.ArrayBuffer.fill(fileMetas.size)(JLong.valueOf(0L))
+ lengths ++= fileMetas.map(file => JLong.valueOf(file.fileSize()))
+ partitionColumns ++= mutable.ArrayBuffer.fill(fileMetas.size)(partitionRow)
+ buckets ++= fileMetas.map(_ => JInteger.valueOf(bucket))
+ firstRowIds ++= fileMetas
+ .map(_.firstRowId())
+ .map(id => JLong.valueOf(if (id == null) 0L else id.toLong))
+ maxSequenceNumbers ++= fileMetas.map(
+ file => JLong.valueOf(file.maxSequenceNumber()))
+ splitGroups ++= fileMetas.map(_ => JInteger.valueOf(splitIdx))
+ case (other, _) =>
+ throw new UnsupportedOperationException(
+ s"paimon split type: '${other.getClass.getName}' is not supported")
}
- case o =>
- throw new GlutenNotSupportException(s"Unsupported input partition type: $o")
+ case other =>
+ throw new GlutenNotSupportException(s"Unsupported input partition type: $other")
}
+ val preferredLoc =
+ SoftAffinity.getFilePartitionLocations(paths.toArray, partition.preferredLocations())
PaimonLocalFilesBuilder.makePaimonLocalFiles(
- p.index,
+ partition.index,
paths.asJava,
starts.asJava,
lengths.asJava,
partitionColumns.asJava,
fileFormat,
- SoftAffinity
- .getFilePartitionLocations(paths.toArray, p.preferredLocations())
- .toList
- .asJava,
- new JHashMap[String, String]()
+ preferredLoc.toList.asJava,
+ new JHashMap[String, String](),
+ buckets.asJava,
+ firstRowIds.asJava,
+ maxSequenceNumbers.asJava,
+ splitGroups.asJava,
+ useHiveSplit,
+ primaryKeys,
+ allRawConvertible
)
- case _ => throw new GlutenNotSupportException()
+ case (other, _) =>
+ throw new GlutenNotSupportException(s"Unsupported input partition type: $other")
}
}
- override def doCanonicalize(): PaimonScanTransformer = {
- this.copy(
- output = output.map(QueryPlan.normalizeExpressions(_, output)),
- runtimeFilters = QueryPlan.normalizePredicates(
- runtimeFilters.filterNot(_ == DynamicPruningExpression(Literal.TrueLiteral)),
- output),
- pushDownFilters = pushDownFilters.map(QueryPlan.normalizePredicates(_, output))
- )
- }
-
override protected[this] def supportsBatchScan(scan: Scan): Boolean =
- PaimonScanTransformer.supportsBatchScan(scan)
+ AbstractPaimonScanTransformer.supportsBatchScan(scan).ok()
}
-object PaimonScanTransformer {
+object AbstractPaimonScanTransformer {
def apply(batchScan: BatchScanExec): PaimonScanTransformer = {
- new PaimonScanTransformer(
+ PaimonScanTransformer(
batchScan.output,
batchScan.scan,
batchScan.runtimeFilters,
@@ -208,23 +243,105 @@ object PaimonScanTransformer {
)
}
- private def getRowDataPartitionComputer(scan: Scan): InternalRowPartitionComputer = {
+ def supportsBatchScan(scan: Scan): ValidationResult = {
+ if (!SQLConf.get.getConf(PaimonConfig.PAIMON_NATIVE_SOURCE_ENABLED)) {
+ return ValidationResult.failed("[Paimon Fallback]: The paimon native source is not enabled.")
+ }
+
scan match {
case paimonScan: PaimonScan =>
- val table = paimonScan.table.asInstanceOf[FileStoreTable]
- // use __HIVE_DEFAULT_PARTITION__ because velox using this
- new InternalRowPartitionComputer(
- ExternalCatalogUtils.DEFAULT_PARTITION_NAME,
- table.schema().logicalPartitionType(),
- table.partitionKeys.asScala.toArray,
- false
+ val table = paimonScan.table
+ val coreOptions = new CoreOptions(table.options())
+ val partitionKeys = table.partitionKeys()
+ val partitionType = table.rowType().project(partitionKeys)
+
+ if (!table.isInstanceOf[FileStoreTable]) {
+ return ValidationResult.failed(
+ s"[Paimon Fallback]: The table is not fileStoreTable: ${table.getClass}")
+ }
+
+ if (partitionType.getFieldTypes.toArray.exists(_.isInstanceOf[DecimalType])) {
+ return ValidationResult.failed(
+ "[Paimon Fallback]: Not support decimal type as partition column")
+ }
+
+ val formatString = coreOptions.fileFormatString()
+ if (
+ !formatString.equalsIgnoreCase(CoreOptions.FILE_FORMAT_PARQUET) &&
+ !formatString.equalsIgnoreCase(CoreOptions.FILE_FORMAT_ORC)
+ ) {
+ return ValidationResult.failed(
+ "[Paimon Fallback]: Only support parquet/orc Paimon table.")
+ }
+
+ val isAllParquet = table
+ .asInstanceOf[FileStoreTable]
+ .coreOptions()
+ .fileFormatPerLevel()
+ .values()
+ .stream()
+ .allMatch(fmt => fmt.equalsIgnoreCase(CoreOptions.FILE_FORMAT_PARQUET)) ||
+ paimonScan.coreOptions.fileFormatString().equalsIgnoreCase(
+ CoreOptions.FILE_FORMAT_PARQUET)
+ val schemaCols = scan.readSchema().fields.map(_.name).toSet
+ val schemaHasMetadataCols = SUPPORTED_METADATA_COLUMNS.exists(schemaCols.contains)
+ if (schemaHasMetadataCols && !isAllParquet) {
+ return ValidationResult.failed(
+ "[Paimon Fallback]: Metadata column queries are only supported with parquet files.")
+ }
+
+ val allSplitsRawConvertible = paimonScan.toBatch.planInputPartitions().forall {
+ case partition: PaimonInputPartition =>
+ partition.splits.forall(_.convertToRawFiles().isPresent)
+ case _ => false
+ }
+ BackendsApiManager.getSparkPlanExecApiInstance.validatePaimonScanCapabilities(
+ hasPrimaryKeys = !table.primaryKeys().isEmpty,
+ allSplitsRawConvertible = allSplitsRawConvertible,
+ deletionVectorsEnabled = coreOptions.deletionVectorsEnabled(),
+ changelogProducer = coreOptions.changelogProducer().name(),
+ mergeEngine = coreOptions.mergeEngine().name()
)
- case _ =>
- throw new GlutenNotSupportException("Only support PaimonScan.")
+ case other =>
+ ValidationResult.failed(
+ s"[Paimon Fallback] Scan is not a PaimonScan. Got ${other.getClass}")
}
}
+}
- def supportsBatchScan(scan: Scan): Boolean = {
- scan.getClass == classOf[PaimonScan]
+case class PaimonScanTransformer(
+ override val output: Seq[AttributeReference],
+ @transient override val scan: Scan,
+ override val runtimeFilters: Seq[Expression],
+ @transient override val table: Table,
+ override val keyGroupedPartitioning: Option[Seq[Expression]] = None,
+ override val commonPartitionValues: Option[Seq[(InternalRow, Int)]] = None,
+ override val pushDownFilters: Option[Seq[Expression]] = None)
+ extends AbstractPaimonScanTransformer(
+ output = output,
+ scan = scan,
+ runtimeFilters = runtimeFilters,
+ table = table,
+ keyGroupedPartitioning = keyGroupedPartitioning,
+ commonPartitionValues = commonPartitionValues,
+ pushDownFilters = pushDownFilters
+ ) {
+
+ override def withNewPushdownFilters(filters: Seq[Expression]): PaimonScanTransformer = {
+ copy(pushDownFilters = Some(filters))
+ }
+
+ override def withOutput(newOutput: Seq[AttributeReference]): BatchScanExecTransformerBase = {
+ copy(output = newOutput)
+ }
+
+ override def doCanonicalize(): PaimonScanTransformer = {
+ copy(
+ output = output.map(QueryPlan.normalizeExpressions(_, output)),
+ runtimeFilters = QueryPlan.normalizePredicates(
+ runtimeFilters.filterNot(_ == DynamicPruningExpression(Literal.TrueLiteral)),
+ output),
+ pushDownFilters = pushDownFilters.map(QueryPlan.normalizePredicates(_, output))
+ )
}
}
diff --git a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonSparkShim.scala
similarity index 60%
copy from gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala
copy to gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonSparkShim.scala
index 19c0c22433..135cebe180 100644
--- a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/OffloadPaimonScan.scala
+++ b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonSparkShim.scala
@@ -16,15 +16,20 @@
*/
package org.apache.gluten.execution
-import org.apache.gluten.extension.columnar.offload.OffloadSingleNode
+import org.apache.spark.internal.Logging
-import org.apache.spark.sql.execution.SparkPlan
-import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.paimon.io.DataFileMeta
+import org.apache.paimon.spark.PaimonScan
+import org.apache.paimon.table.source.DataSplit
+import org.apache.paimon.utils.InternalRowPartitionComputer
-case class OffloadPaimonScan() extends OffloadSingleNode {
- override def offload(plan: SparkPlan): SparkPlan = plan match {
- case scan: BatchScanExec if PaimonScanTransformer.supportsBatchScan(scan.scan) =>
- PaimonScanTransformer(scan)
- case other => other
- }
+trait PaimonSparkShim extends Logging {
+
+ def isChainSplit(split: DataSplit): Boolean
+
+ def getSplitPartition(split: DataSplit): org.apache.paimon.data.InternalRow
+
+ def getBucketPath(split: DataSplit, file: DataFileMeta): String
+
+ def getInternalPartitionComputer(scan: PaimonScan): InternalRowPartitionComputer
}
diff --git a/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonSparkShimImpl.scala b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonSparkShimImpl.scala
new file mode 100644
index 0000000000..fdcc009544
--- /dev/null
+++ b/gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonSparkShimImpl.scala
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils
+
+import org.apache.paimon.data.InternalRow
+import org.apache.paimon.io.DataFileMeta
+import org.apache.paimon.spark.PaimonScan
+import org.apache.paimon.table.FileStoreTable
+import org.apache.paimon.table.source.DataSplit
+import org.apache.paimon.utils.InternalRowPartitionComputer
+
+import scala.collection.JavaConverters._
+
+class PaimonSparkShimImpl extends PaimonSparkShim {
+
+ override def isChainSplit(split: DataSplit): Boolean = false
+
+ override def getSplitPartition(split: DataSplit): InternalRow = split.partition()
+
+ override def getBucketPath(split: DataSplit, file: DataFileMeta): String = split.bucketPath()
+
+ override def getInternalPartitionComputer(scan: PaimonScan): InternalRowPartitionComputer = {
+ val table = scan.table.asInstanceOf[FileStoreTable]
+ new InternalRowPartitionComputer(
+ ExternalCatalogUtils.DEFAULT_PARTITION_NAME,
+ table.schema().logicalPartitionType(),
+ table.partitionKeys().asScala.toArray,
+ false)
+ }
+}
diff --git a/gluten-paimon/src-paimon/main/scala/org/apache/paimon/spark/source/PaimonConfig.scala b/gluten-paimon/src-paimon/main/scala/org/apache/paimon/spark/source/PaimonConfig.scala
new file mode 100644
index 0000000000..e4fcfd7d81
--- /dev/null
+++ b/gluten-paimon/src-paimon/main/scala/org/apache/paimon/spark/source/PaimonConfig.scala
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.paimon.spark.source
+
+import org.apache.spark.sql.internal.SQLConf.buildConf
+
+object PaimonConfig {
+
+ val PAIMON_NATIVE_SOURCE_ENABLED =
+ buildConf("spark.gluten.paimon.native.source.enabled")
+ .doc("When true, enable the paimon native source.")
+ .internal()
+ .booleanConf
+ .createWithDefault(true)
+
+ val PAIMON_NATIVE_MOR_ENABLED =
+ buildConf("spark.gluten.paimon.native.mor.source.enabled")
+ .doc("When true, enables the paimon mor source.")
+ .internal()
+ .booleanConf
+ .createWithDefault(false)
+
+ val PAIMON_NATIVE_MOR_AGGREGATE_ENGINE_ENABLED =
+ buildConf("spark.gluten.paimon.native.mor.aggregate.engine.enabled")
+ .doc("When true, enables the paimon mor aggregate engine.")
+ .internal()
+ .booleanConf
+ .createWithDefault(false)
+
+ val PAIMON_NATIVE_MOR_PARTIAL_UPDATE_ENGINE_ENABLED =
+ buildConf("spark.gluten.paimon.native.mor.partial.update.engine.enabled")
+ .doc("When true, enables the paimon mor partial update engine.")
+ .internal()
+ .booleanConf
+ .createWithDefault(false)
+
+ val PAIMON_NATIVE_SPLIT_ENABLED =
+ buildConf("spark.gluten.paimon.native.split.enabled")
+ .doc("When true, enables the paimon split creation.")
+ .internal()
+ .booleanConf
+ .createWithDefault(true)
+}
diff --git a/gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto b/gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto
index 02c7f4cc5c..5333aab35f 100644
--- a/gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto
+++ b/gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto
@@ -197,6 +197,20 @@ message ReadRel {
repeated DeleteFile delete_files = 3;
}
+ message PaimonReadOptions {
+ int32 bucket = 1;
+ int64 first_row_id = 2;
+ int64 max_sequence_number = 3;
+ int32 split_group = 4;
+ oneof file_format {
+ ParquetReadOptions parquet = 5;
+ OrcReadOptions orc = 6;
+ }
+ bool use_hive_split = 7;
+ bool raw_convertible = 8;
+ repeated string primary_keys = 9;
+ }
+
message DeltaReadOptions {
enum RowIndexFilterType {
KEEP_ALL = 0;
@@ -220,6 +234,7 @@ message ReadRel {
JsonReadOptions json = 15;
IcebergReadOptions iceberg = 16;
DeltaReadOptions delta = 22;
+ PaimonReadOptions paimon = 23;
}
message partitionColumn {
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
index 6aa2f7f974..67a042f020 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
@@ -578,6 +578,27 @@ trait SparkPlanExecApi {
def supportPushDownFilterToScan(sparkExecNode: LeafExecNode): Boolean = true
+ def validatePaimonScanCapabilities(
+ hasPrimaryKeys: Boolean,
+ allSplitsRawConvertible: Boolean,
+ deletionVectorsEnabled: Boolean,
+ changelogProducer: String,
+ mergeEngine: String): ValidationResult = {
+ if (deletionVectorsEnabled) {
+ return ValidationResult.failed(
+ "[Paimon Fallback]: The scan with deletion vector is not supported")
+ }
+ if (hasPrimaryKeys) {
+ return ValidationResult.failed("[Paimon Fallback]: Not support Paimon PK table.")
+ }
+ ValidationResult.succeeded
+ }
+
+ def rewritePaimonPushdownFilters(
+ filters: Seq[Expression],
+ primaryKeys: Set[String],
+ metadataColumns: Set[String]): Seq[Expression] = filters
+
/** Return whether the filter is supported in scan. */
def isSupportedScanFilter(filter: Expression, sparkExecNode: LeafExecNode): Boolean = {
ExpressionConverter.canReplaceWithExpressionTransformer(
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala
index 6d265100d0..d830e3fc8e 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala
@@ -37,9 +37,11 @@ import scala.collection.JavaConverters._
trait BasicScanExecTransformer extends LeafTransformSupport with BaseDataSource {
import org.apache.spark.sql.catalyst.util._
+ protected def rewriteNativeScanFilters(filters: Seq[Expression]): Seq[Expression] = filters
+
/** Returns the filters that can be pushed down to native file scan */
final def filterExprs(): Seq[Expression] = {
- if (pushDownFilters.nonEmpty) {
+ val nativeFilters = if (pushDownFilters.nonEmpty) {
val (_, scanFiltersNotInPushDownFilters) =
scanFilters.partition(pushDownFilters.get.contains(_))
// For filters that only exists in scan, we need to check if they are supported.
@@ -59,6 +61,7 @@ trait BasicScanExecTransformer extends LeafTransformSupport with BaseDataSource
scanFilters.filter(
BackendsApiManager.getSparkPlanExecApiInstance.isSupportedScanFilter(_, this))
}
+ rewriteNativeScanFilters(nativeFilters)
}
/** Returns the filters that already exists in scan. */
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 20/38: [GLUTEN][CORE] Share native write test checker base
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 151f2420dc1a1487073014a29673d425e2a42098
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 22:07:47 2026 +0800
[GLUTEN][CORE] Share native write test checker base
Extract the common native-write listener and assertion flow into backends-core so backend test suites only keep their plan normalization and native-plan detection hooks.
---
.../apache/spark/gluten/NativeWriteChecker.scala | 53 ++++++++---------
.../gluten/test/NativeWriteCheckerBase.scala | 69 ++++++++++++++++++++++
.../apache/spark/sql/execution/WriteUtils.scala | 38 ++++--------
3 files changed, 104 insertions(+), 56 deletions(-)
diff --git a/backends-clickhouse/src/test/scala/org/apache/spark/gluten/NativeWriteChecker.scala b/backends-clickhouse/src/test/scala/org/apache/spark/gluten/NativeWriteChecker.scala
index 49168988fa..416debaab8 100644
--- a/backends-clickhouse/src/test/scala/org/apache/spark/gluten/NativeWriteChecker.scala
+++ b/backends-clickhouse/src/test/scala/org/apache/spark/gluten/NativeWriteChecker.scala
@@ -18,11 +18,11 @@ package org.apache.spark.gluten
import org.apache.gluten.config.GlutenConfig
import org.apache.gluten.execution.{CHColumnarToCarrierRowExec, GlutenClickHouseWholeStageTransformerSuite}
+import org.apache.gluten.test.NativeWriteCheckerBase
import org.apache.spark.sql.{Dataset, Row}
-import org.apache.spark.sql.execution.{ColumnarWriteFilesExec, QueryExecution}
+import org.apache.spark.sql.execution.{ColumnarWriteFilesExec, QueryExecution, SparkPlan}
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
-import org.apache.spark.sql.util.QueryExecutionListener
import java.io.File
@@ -30,6 +30,7 @@ import scala.reflect.runtime.universe.TypeTag
trait NativeWriteChecker
extends GlutenClickHouseWholeStageTransformerSuite
+ with NativeWriteCheckerBase
with AdaptiveSparkPlanHelper {
private val formats: Seq[String] = Seq("orc", "parquet")
@@ -39,37 +40,31 @@ trait NativeWriteChecker
protected def getWarehouseDir: String = dataHome + "/中文/spark-warehouse"
// scalastyle:on nonascii
- def withNativeWriteCheck(checkNative: Boolean)(block: => Unit): Unit = {
- var nativeUsed = false
+ override protected def waitUntilListenerBusEmpty(): Unit = {
+ spark.sparkContext.listenerBus.waitUntilEmpty()
+ }
- val queryListener = new QueryExecutionListener {
- override def onFailure(f: String, qe: QueryExecution, e: Exception): Unit = {
- fail("query failed", e)
- }
- override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = {
- if (!nativeUsed) {
- val executedPlan = stripAQEPlan(qe.executedPlan)
- nativeUsed = if (isSparkVersionGE("3.5")) {
- executedPlan.find(_.isInstanceOf[ColumnarWriteFilesExec]).isDefined
- } else {
- executedPlan.find(_.isInstanceOf[CHColumnarToCarrierRowExec]).isDefined
- }
- }
- }
- }
- try {
- spark.listenerManager.register(queryListener)
- block
- spark.sparkContext.listenerBus.waitUntilEmpty()
- assertResult(checkNative)(nativeUsed)
- } finally {
- spark.listenerManager.unregister(queryListener)
+ override protected def normalizeExecutedPlan(qe: QueryExecution): SparkPlan = {
+ stripAQEPlan(qe.executedPlan)
+ }
+
+ override protected def isNativeWritePlan(plan: SparkPlan): Boolean = {
+ if (isSparkVersionGE("3.5")) {
+ plan.find(_.isInstanceOf[ColumnarWriteFilesExec]).isDefined
+ } else {
+ plan.find(_.isInstanceOf[CHColumnarToCarrierRowExec]).isDefined
}
}
+
+ override protected def onNativeWriteQueryFailure(
+ funcName: String,
+ qe: QueryExecution,
+ error: Exception): Unit = {
+ fail("query failed", error)
+ }
+
def checkInsertQuery(sqlStr: String, checkNative: Boolean): Unit =
- withNativeWriteCheck(checkNative) {
- spark.sql(sqlStr)
- }
+ checkNativeWrite(sqlStr, checkNative)
def withDestinationTable(table: String, createTableSql: Option[String] = None)(
f: => Unit): Unit = {
diff --git a/backends-core/src/main/scala/org/apache/gluten/test/NativeWriteCheckerBase.scala b/backends-core/src/main/scala/org/apache/gluten/test/NativeWriteCheckerBase.scala
new file mode 100644
index 0000000000..6e8507b44e
--- /dev/null
+++ b/backends-core/src/main/scala/org/apache/gluten/test/NativeWriteCheckerBase.scala
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.test
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.execution.{QueryExecution, SparkPlan}
+import org.apache.spark.sql.util.QueryExecutionListener
+
+trait NativeWriteCheckerBase {
+
+ protected def spark: SparkSession
+
+ protected def normalizeExecutedPlan(qe: QueryExecution): SparkPlan = qe.executedPlan
+
+ protected def isNativeWritePlan(plan: SparkPlan): Boolean
+
+ protected def waitUntilListenerBusEmpty(): Unit
+
+ protected def onNativeWriteQueryFailure(
+ funcName: String,
+ qe: QueryExecution,
+ error: Exception): Unit = {}
+
+ protected def withNativeWriteCheck(expectedNativeWrite: Boolean)(block: => Unit): Unit = {
+ var nativeUsed = false
+ val queryListener = new QueryExecutionListener {
+ override def onFailure(funcName: String, qe: QueryExecution, e: Exception): Unit = {
+ onNativeWriteQueryFailure(funcName, qe, e)
+ }
+
+ override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = {
+ if (!nativeUsed) {
+ nativeUsed = isNativeWritePlan(normalizeExecutedPlan(qe))
+ }
+ }
+ }
+
+ try {
+ spark.listenerManager.register(queryListener)
+ block
+ waitUntilListenerBusEmpty()
+ assert(
+ nativeUsed == expectedNativeWrite,
+ s"Expected native write usage to be $expectedNativeWrite, but was $nativeUsed")
+ } finally {
+ spark.listenerManager.unregister(queryListener)
+ }
+ }
+
+ protected def checkNativeWrite(sqlStr: String, expectNative: Boolean = true): Unit = {
+ withNativeWriteCheck(expectNative) {
+ spark.sql(sqlStr)
+ }
+ }
+}
diff --git a/backends-velox/src/test/scala/org/apache/spark/sql/execution/WriteUtils.scala b/backends-velox/src/test/scala/org/apache/spark/sql/execution/WriteUtils.scala
index 416ce41f57..6789c6088b 100644
--- a/backends-velox/src/test/scala/org/apache/spark/sql/execution/WriteUtils.scala
+++ b/backends-velox/src/test/scala/org/apache/spark/sql/execution/WriteUtils.scala
@@ -17,47 +17,31 @@
package org.apache.spark.sql.execution
import org.apache.gluten.execution.VeloxColumnarToCarrierRowExec
+import org.apache.gluten.test.NativeWriteCheckerBase
import org.apache.spark.sql.{DataFrame, GlutenQueryTest}
import org.apache.spark.sql.catalyst.expressions.{BitwiseAnd, Expression, HiveHash, Literal, Pmod, UnsafeProjection}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.test.SQLTestUtils
-import org.apache.spark.sql.util.QueryExecutionListener
import java.io.File
-trait WriteUtils extends GlutenQueryTest with SQLTestUtils {
+trait WriteUtils extends GlutenQueryTest with SQLTestUtils with NativeWriteCheckerBase {
+
+ override protected def waitUntilListenerBusEmpty(): Unit = {
+ spark.sparkContext.listenerBus.waitUntilEmpty()
+ }
def tableDir(table: String): File = {
val identifier = spark.sessionState.sqlParser.parseTableIdentifier(table)
new File(spark.sessionState.catalog.defaultTablePath(identifier))
}
- def checkNativeWrite(sqlStr: String, expectNative: Boolean = true): Unit = {
- var nativeUsed = false
- val queryListener = new QueryExecutionListener {
- override def onFailure(f: String, qe: QueryExecution, e: Exception): Unit = {}
- override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = {
- if (!nativeUsed) {
- nativeUsed = if (isSparkVersionGE("3.4")) {
- qe.executedPlan.exists(_.isInstanceOf[ColumnarWriteFilesExec])
- } else {
- qe.executedPlan.exists(_.isInstanceOf[VeloxColumnarToCarrierRowExec])
- }
- }
- }
- }
- try {
- spark.listenerManager.register(queryListener)
- spark.sql(sqlStr)
- spark.sparkContext.listenerBus.waitUntilEmpty()
- if (expectNative) {
- assert(nativeUsed)
- } else {
- assert(!nativeUsed)
- }
- } finally {
- spark.listenerManager.unregister(queryListener)
+ override protected def isNativeWritePlan(plan: SparkPlan): Boolean = {
+ if (isSparkVersionGE("3.4")) {
+ plan.exists(_.isInstanceOf[ColumnarWriteFilesExec])
+ } else {
+ plan.exists(_.isInstanceOf[VeloxColumnarToCarrierRowExec])
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 09/38: [GLUTEN][CORE] Move benchmark data to shared directory
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit aa18e3ba2255be4b34c805820c5e449f87bbe517
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 20:09:47 2026 +0800
[GLUTEN][CORE] Move benchmark data to shared directory
Share the Velox benchmark input data under a backend-neutral path so future backends can reuse the same files without duplicating references.
---
.../org/apache/gluten/execution/MiscOperatorSuite.scala | 2 +-
.../benchmarks/data/bm_lineitem/orc/lineitem.orc | Bin
.../benchmarks/data/bm_lineitem/orc/long_decimal_nonull.orc | Bin
...000-482d0e2c-7667-42bb-8750-64ecfe331d28-c000.snappy.orc | Bin
.../data/bm_lineitem/orc/short_decimal_nonull.orc | Bin
...3a972d10-5e0b-4bde-b250-032afd81a132-c000.snappy.parquet | Bin
.../benchmarks/data/generic_q1/q1_first_stage_0.json | 0
.../benchmarks/data/generic_q5/q5_first_stage_0.json | 0
.../benchmarks/data/generic_q5/q5_first_stage_0_split.json | 0
.../benchmarks/data/generic_q5/q5_middle_stage_0.json | 0
.../data/generic_q5/q5_middle_stage_0_iter_0.parquet | Bin
.../data/generic_q5/q5_middle_stage_0_iter_1.parquet | Bin
cpp/{velox => }/benchmarks/data/plan/q17_joins.json | 0
cpp/{velox => }/benchmarks/data/plan/select.json | 0
.../benchmarks/data/plan/select_long_decimal.json | 0
.../benchmarks/data/plan/select_short_decimal.json | 0
...a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet | Bin
...6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet | Bin
...8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet | Bin
...41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet | Bin
...e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet | Bin
...2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet | Bin
...90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet | Bin
...0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet | Bin
docs/developers/MicroBenchmarks.md | 12 ++++++------
25 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala b/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala
index fcf1fbbbf7..63650844c6 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala
+++ b/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala
@@ -735,7 +735,7 @@ class MiscOperatorSuite extends VeloxWholeStageTransformerSuite with AdaptiveSpa
test("orc scan") {
val df = spark.read
.format("orc")
- .load("../cpp/velox/benchmarks/data/bm_lineitem/orc/lineitem.orc")
+ .load("../../cpp/benchmarks/data/bm_lineitem/orc/lineitem.orc")
df.createOrReplaceTempView("lineitem_orc")
runQueryAndCompare("select l_orderkey from lineitem_orc") {
df =>
diff --git a/cpp/velox/benchmarks/data/bm_lineitem/orc/lineitem.orc b/cpp/benchmarks/data/bm_lineitem/orc/lineitem.orc
similarity index 100%
rename from cpp/velox/benchmarks/data/bm_lineitem/orc/lineitem.orc
rename to cpp/benchmarks/data/bm_lineitem/orc/lineitem.orc
diff --git a/cpp/velox/benchmarks/data/bm_lineitem/orc/long_decimal_nonull.orc b/cpp/benchmarks/data/bm_lineitem/orc/long_decimal_nonull.orc
similarity index 100%
rename from cpp/velox/benchmarks/data/bm_lineitem/orc/long_decimal_nonull.orc
rename to cpp/benchmarks/data/bm_lineitem/orc/long_decimal_nonull.orc
diff --git a/cpp/velox/benchmarks/data/bm_lineitem/orc/part-00000-482d0e2c-7667-42bb-8750-64ecfe331d28-c000.snappy.orc b/cpp/benchmarks/data/bm_lineitem/orc/part-00000-482d0e2c-7667-42bb-8750-64ecfe331d28-c000.snappy.orc
similarity index 100%
rename from cpp/velox/benchmarks/data/bm_lineitem/orc/part-00000-482d0e2c-7667-42bb-8750-64ecfe331d28-c000.snappy.orc
rename to cpp/benchmarks/data/bm_lineitem/orc/part-00000-482d0e2c-7667-42bb-8750-64ecfe331d28-c000.snappy.orc
diff --git a/cpp/velox/benchmarks/data/bm_lineitem/orc/short_decimal_nonull.orc b/cpp/benchmarks/data/bm_lineitem/orc/short_decimal_nonull.orc
similarity index 100%
rename from cpp/velox/benchmarks/data/bm_lineitem/orc/short_decimal_nonull.orc
rename to cpp/benchmarks/data/bm_lineitem/orc/short_decimal_nonull.orc
diff --git a/cpp/velox/benchmarks/data/bm_lineitem/parquet/part-00287-3a972d10-5e0b-4bde-b250-032afd81a132-c000.snappy.parquet b/cpp/benchmarks/data/bm_lineitem/parquet/part-00287-3a972d10-5e0b-4bde-b250-032afd81a132-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/bm_lineitem/parquet/part-00287-3a972d10-5e0b-4bde-b250-032afd81a132-c000.snappy.parquet
rename to cpp/benchmarks/data/bm_lineitem/parquet/part-00287-3a972d10-5e0b-4bde-b250-032afd81a132-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/generic_q1/q1_first_stage_0.json b/cpp/benchmarks/data/generic_q1/q1_first_stage_0.json
similarity index 100%
rename from cpp/velox/benchmarks/data/generic_q1/q1_first_stage_0.json
rename to cpp/benchmarks/data/generic_q1/q1_first_stage_0.json
diff --git a/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0.json b/cpp/benchmarks/data/generic_q5/q5_first_stage_0.json
similarity index 100%
rename from cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0.json
rename to cpp/benchmarks/data/generic_q5/q5_first_stage_0.json
diff --git a/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json b/cpp/benchmarks/data/generic_q5/q5_first_stage_0_split.json
similarity index 100%
rename from cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json
rename to cpp/benchmarks/data/generic_q5/q5_first_stage_0_split.json
diff --git a/cpp/velox/benchmarks/data/generic_q5/q5_middle_stage_0.json b/cpp/benchmarks/data/generic_q5/q5_middle_stage_0.json
similarity index 100%
rename from cpp/velox/benchmarks/data/generic_q5/q5_middle_stage_0.json
rename to cpp/benchmarks/data/generic_q5/q5_middle_stage_0.json
diff --git a/cpp/velox/benchmarks/data/generic_q5/q5_middle_stage_0_iter_0.parquet b/cpp/benchmarks/data/generic_q5/q5_middle_stage_0_iter_0.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/generic_q5/q5_middle_stage_0_iter_0.parquet
rename to cpp/benchmarks/data/generic_q5/q5_middle_stage_0_iter_0.parquet
diff --git a/cpp/velox/benchmarks/data/generic_q5/q5_middle_stage_0_iter_1.parquet b/cpp/benchmarks/data/generic_q5/q5_middle_stage_0_iter_1.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/generic_q5/q5_middle_stage_0_iter_1.parquet
rename to cpp/benchmarks/data/generic_q5/q5_middle_stage_0_iter_1.parquet
diff --git a/cpp/velox/benchmarks/data/plan/q17_joins.json b/cpp/benchmarks/data/plan/q17_joins.json
similarity index 100%
rename from cpp/velox/benchmarks/data/plan/q17_joins.json
rename to cpp/benchmarks/data/plan/q17_joins.json
diff --git a/cpp/velox/benchmarks/data/plan/select.json b/cpp/benchmarks/data/plan/select.json
similarity index 100%
rename from cpp/velox/benchmarks/data/plan/select.json
rename to cpp/benchmarks/data/plan/select.json
diff --git a/cpp/velox/benchmarks/data/plan/select_long_decimal.json b/cpp/benchmarks/data/plan/select_long_decimal.json
similarity index 100%
rename from cpp/velox/benchmarks/data/plan/select_long_decimal.json
rename to cpp/benchmarks/data/plan/select_long_decimal.json
diff --git a/cpp/velox/benchmarks/data/plan/select_short_decimal.json b/cpp/benchmarks/data/plan/select_short_decimal.json
similarity index 100%
rename from cpp/velox/benchmarks/data/plan/select_short_decimal.json
rename to cpp/benchmarks/data/plan/select_short_decimal.json
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet
diff --git a/cpp/velox/benchmarks/data/tpch_sf10m/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet b/cpp/benchmarks/data/tpch_sf10m/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet
similarity index 100%
rename from cpp/velox/benchmarks/data/tpch_sf10m/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet
rename to cpp/benchmarks/data/tpch_sf10m/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet
diff --git a/docs/developers/MicroBenchmarks.md b/docs/developers/MicroBenchmarks.md
index ce0996a6cc..97edb9b648 100644
--- a/docs/developers/MicroBenchmarks.md
+++ b/docs/developers/MicroBenchmarks.md
@@ -426,24 +426,24 @@ done
### Run Examples
-We also provide some example inputs in [cpp/velox/benchmarks/data](../../cpp/velox/benchmarks/data).
-E.g. Files under [generic_q5](../../cpp/velox/benchmarks/data/generic_q5) simulates a first-stage in
+We also provide some example inputs in [cpp/benchmarks/data](../../cpp/benchmarks/data).
+E.g. Files under [generic_q5](../../cpp/benchmarks/data/generic_q5) simulates a first-stage in
TPCH Q5, which has a heavy table scan. You can follow below steps to run this example.
1.
-Open [generic_q5/q5_first_stage_0_split.json](../../cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json)
+Open [generic_q5/q5_first_stage_0_split.json](../../cpp/benchmarks/data/generic_q5/q5_first_stage_0_split.json)
with file editor. Search for `"uriFile": "LINEITEM"` and replace `LINEITEM` with the URI to one
partition file in lineitem. In the next line, replace the number in `"length": "..."` with the
actual file length. Suppose you are using the provided small TPCH table
-in [cpp/velox/benchmarks/data/tpch_sf10m](../../cpp/velox/benchmarks/data/tpch_sf10m), the replaced
+in [cpp/benchmarks/data/tpch_sf10m](../../cpp/benchmarks/data/tpch_sf10m), the replaced
JSON should be like:
```
{
"items": [
{
- "uriFile": "file:///path/to/gluten/cpp/velox/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet",
+ "uriFile": "file:///path/to/gluten/cpp/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet",
"length": "1863237",
"parquet": {}
}
@@ -462,7 +462,7 @@ processes=24 # Same value of spark.executor.instances
threads=8 # Same value of spark.executor.cores
for ((i=0; i<${processes}; i++)); do
- ./generic_benchmark --plan /path/to/gluten/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0.json --split /path/to/gluten/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json --noprint-result --with-shuffle --threads $threads --cpu $((i*threads)) &
+ ./generic_benchmark --plan /path/to/gluten/cpp/benchmarks/data/generic_q5/q5_first_stage_0.json --split /path/to/gluten/cpp/benchmarks/data/generic_q5/q5_first_stage_0_split.json --noprint-result --with-shuffle --threads $threads --cpu $((i*threads)) &
done >stdout.log 2>stderr.log
```
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 25/38: [GLUTEN][CORE] Share JVM tier-1 test scaffolding under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit f89483359a2f603fc869d76ab053e37e937f3014
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 18:02:57 2026 +0800
[GLUTEN][CORE] Share JVM tier-1 test scaffolding under backends-core
Open up a test source root in backends-core and lift the velox-side test
files that are byte-identical across bolt/velox (after normalising
Bolt/Velox naming) so both backends can share them via the backends-core
test-jar. Files lifted in this patch:
- fs: ArrowFilesystemTest, CsvWriteSupport, TestDataset, TestNativeDataset
- tags: EnhancedFeaturesTest, FuzzerTest, SkipTest, UDFTest
- udf/udtf: CustomerUDF, ConditionalOutputUDTF, CustomerUDTF,
NoInputUDTF, SimpleUDTF
- misc: GlutenSQLCollectTailExecSuite, FuzzerResult,
StreamingAggregateBenchmark
Adds scalatest/junit/spark test-jar/arrow-dataset test dependencies and a
test-jar build to backends-core, and wires backends-velox to depend on
backends-core test-jar so the relocated tests run from the shared module.
---
backends-core/pom.xml | 65 ++++++++++++++++++++++
.../org/apache/gluten/fs/ArrowFilesystemTest.java | 0
.../java/org/apache/gluten/fs/CsvWriteSupport.java | 0
.../java/org/apache/gluten/fs/TestDataset.java | 0
.../org/apache/gluten/fs/TestNativeDataset.java | 0
.../apache/gluten/tags/EnhancedFeaturesTest.java | 0
.../java/org/apache/gluten/tags/FuzzerTest.java | 0
.../test/java/org/apache/gluten/tags/SkipTest.java | 0
.../test/java/org/apache/gluten/tags/UDFTest.java | 0
.../java/org/apache/gluten/udf/CustomerUDF.java | 0
.../apache/gluten/udtf/ConditionalOutputUDTF.java | 0
.../java/org/apache/gluten/udtf/CustomerUDTF.java | 0
.../java/org/apache/gluten/udtf/NoInputUDTF.java | 0
.../java/org/apache/gluten/udtf/SimpleUDTF.java | 0
.../execution/GlutenSQLCollectTailExecSuite.scala | 0
.../org/apache/gluten/fuzzer/FuzzerResult.scala | 0
.../benchmark/StreamingAggregateBenchmark.scala | 0
backends-velox/pom.xml | 7 +++
18 files changed, 72 insertions(+)
diff --git a/backends-core/pom.xml b/backends-core/pom.xml
index 8903c9a45a..22a445eb27 100644
--- a/backends-core/pom.xml
+++ b/backends-core/pom.xml
@@ -56,6 +56,58 @@
<version>${scala.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-substrait</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-hive_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-dataset</artifactId>
+ <version>${arrow.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.arrow</groupId>
+ <artifactId>arrow-vector</artifactId>
+ <version>${arrow.version}</version>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<build>
@@ -83,6 +135,19 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-test-jar</id>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <version>${scalatest-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
diff --git a/backends-velox/src/test/java/org/apache/gluten/fs/ArrowFilesystemTest.java b/backends-core/src/test/java/org/apache/gluten/fs/ArrowFilesystemTest.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/fs/ArrowFilesystemTest.java
rename to backends-core/src/test/java/org/apache/gluten/fs/ArrowFilesystemTest.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/fs/CsvWriteSupport.java b/backends-core/src/test/java/org/apache/gluten/fs/CsvWriteSupport.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/fs/CsvWriteSupport.java
rename to backends-core/src/test/java/org/apache/gluten/fs/CsvWriteSupport.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/fs/TestDataset.java b/backends-core/src/test/java/org/apache/gluten/fs/TestDataset.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/fs/TestDataset.java
rename to backends-core/src/test/java/org/apache/gluten/fs/TestDataset.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/fs/TestNativeDataset.java b/backends-core/src/test/java/org/apache/gluten/fs/TestNativeDataset.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/fs/TestNativeDataset.java
rename to backends-core/src/test/java/org/apache/gluten/fs/TestNativeDataset.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/tags/EnhancedFeaturesTest.java b/backends-core/src/test/java/org/apache/gluten/tags/EnhancedFeaturesTest.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/tags/EnhancedFeaturesTest.java
rename to backends-core/src/test/java/org/apache/gluten/tags/EnhancedFeaturesTest.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/tags/FuzzerTest.java b/backends-core/src/test/java/org/apache/gluten/tags/FuzzerTest.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/tags/FuzzerTest.java
rename to backends-core/src/test/java/org/apache/gluten/tags/FuzzerTest.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/tags/SkipTest.java b/backends-core/src/test/java/org/apache/gluten/tags/SkipTest.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/tags/SkipTest.java
rename to backends-core/src/test/java/org/apache/gluten/tags/SkipTest.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/tags/UDFTest.java b/backends-core/src/test/java/org/apache/gluten/tags/UDFTest.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/tags/UDFTest.java
rename to backends-core/src/test/java/org/apache/gluten/tags/UDFTest.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/udf/CustomerUDF.java b/backends-core/src/test/java/org/apache/gluten/udf/CustomerUDF.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/udf/CustomerUDF.java
rename to backends-core/src/test/java/org/apache/gluten/udf/CustomerUDF.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/udtf/ConditionalOutputUDTF.java b/backends-core/src/test/java/org/apache/gluten/udtf/ConditionalOutputUDTF.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/udtf/ConditionalOutputUDTF.java
rename to backends-core/src/test/java/org/apache/gluten/udtf/ConditionalOutputUDTF.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/udtf/CustomerUDTF.java b/backends-core/src/test/java/org/apache/gluten/udtf/CustomerUDTF.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/udtf/CustomerUDTF.java
rename to backends-core/src/test/java/org/apache/gluten/udtf/CustomerUDTF.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/udtf/NoInputUDTF.java b/backends-core/src/test/java/org/apache/gluten/udtf/NoInputUDTF.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/udtf/NoInputUDTF.java
rename to backends-core/src/test/java/org/apache/gluten/udtf/NoInputUDTF.java
diff --git a/backends-velox/src/test/java/org/apache/gluten/udtf/SimpleUDTF.java b/backends-core/src/test/java/org/apache/gluten/udtf/SimpleUDTF.java
similarity index 100%
rename from backends-velox/src/test/java/org/apache/gluten/udtf/SimpleUDTF.java
rename to backends-core/src/test/java/org/apache/gluten/udtf/SimpleUDTF.java
diff --git a/backends-velox/src/test/scala/org/apache/gluten/execution/GlutenSQLCollectTailExecSuite.scala b/backends-core/src/test/scala/org/apache/gluten/execution/GlutenSQLCollectTailExecSuite.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/execution/GlutenSQLCollectTailExecSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/execution/GlutenSQLCollectTailExecSuite.scala
diff --git a/backends-velox/src/test/scala/org/apache/gluten/fuzzer/FuzzerResult.scala b/backends-core/src/test/scala/org/apache/gluten/fuzzer/FuzzerResult.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/fuzzer/FuzzerResult.scala
rename to backends-core/src/test/scala/org/apache/gluten/fuzzer/FuzzerResult.scala
diff --git a/backends-velox/src/test/scala/org/apache/spark/sql/execution/benchmark/StreamingAggregateBenchmark.scala b/backends-core/src/test/scala/org/apache/spark/sql/execution/benchmark/StreamingAggregateBenchmark.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/spark/sql/execution/benchmark/StreamingAggregateBenchmark.scala
rename to backends-core/src/test/scala/org/apache/spark/sql/execution/benchmark/StreamingAggregateBenchmark.scala
diff --git a/backends-velox/pom.xml b/backends-velox/pom.xml
index 3e81895ea7..392cde2f7d 100644
--- a/backends-velox/pom.xml
+++ b/backends-velox/pom.xml
@@ -46,6 +46,13 @@
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>backends-core</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 35/38: Share carrier row transitions under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit a938159153b63221b91ff8bc19cb571ce7747c10
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Wed Jul 1 16:15:31 2026 +0800
Share carrier row transitions under backends-core
Move the common carrier-row transition glue into backends-core so the velox backend can keep only thin wrappers and future backend alignment can reuse the same shared implementation.
---
.../gluten/execution/SharedCarrierRowType.scala | 15 +++++++++-----
.../execution/SharedColumnarToCarrierRowExec.scala | 24 ++++++++++++----------
.../backendsapi/velox/VeloxCarrierRowType.scala | 12 +++++------
.../execution/VeloxColumnarToCarrierRowExec.scala | 12 +++++------
4 files changed, 34 insertions(+), 29 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala b/backends-core/src/main/scala/org/apache/gluten/execution/SharedCarrierRowType.scala
similarity index 67%
copy from backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala
copy to backends-core/src/main/scala/org/apache/gluten/execution/SharedCarrierRowType.scala
index 79d350e90a..58ce6531bd 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/execution/SharedCarrierRowType.scala
@@ -14,13 +14,18 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.gluten.backendsapi.velox
+package org.apache.gluten.execution
-import org.apache.gluten.execution.VeloxColumnarToCarrierRowExec
-import org.apache.gluten.extension.columnar.transition.Convention
+import org.apache.gluten.extension.columnar.transition.{Convention, Transition}
+
+/** Shared implementation for backend-specific carrier row types. */
+abstract class SharedCarrierRowType extends Convention.RowType {
+
+ protected def backendBatchType: Convention.BatchType
+
+ protected def toCarrierRowTransition: Transition
-object VeloxCarrierRowType extends Convention.RowType {
override protected[this] def registerTransitions(): Unit = {
- fromBatch(VeloxBatchType, VeloxColumnarToCarrierRowExec.apply)
+ fromBatch(backendBatchType, toCarrierRowTransition)
}
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala b/backends-core/src/main/scala/org/apache/gluten/execution/SharedColumnarToCarrierRowExec.scala
similarity index 66%
copy from backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala
copy to backends-core/src/main/scala/org/apache/gluten/execution/SharedColumnarToCarrierRowExec.scala
index 65ce184861..0ca2a505d2 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/execution/SharedColumnarToCarrierRowExec.scala
@@ -16,23 +16,25 @@
*/
package org.apache.gluten.execution
-import org.apache.gluten.backendsapi.velox.{VeloxBatchType, VeloxCarrierRowType}
import org.apache.gluten.extension.columnar.transition.{Convention, ConventionReq, Transitions}
import org.apache.spark.sql.execution.SparkPlan
-case class VeloxColumnarToCarrierRowExec(override val child: SparkPlan)
+/** Shared implementation for backend-specific `ColumnarToCarrierRowExec` wrappers. */
+abstract class SharedColumnarToCarrierRowExec(child: SparkPlan)
extends ColumnarToCarrierRowExecBase {
- override protected def fromBatchType(): Convention.BatchType = VeloxBatchType
- override def rowType(): Convention.RowType = VeloxCarrierRowType
- override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =
- copy(child = newChild)
+
+ protected def backendBatchType: Convention.BatchType
+
+ protected def backendRowType: Convention.RowType
+
+ override protected def fromBatchType(): Convention.BatchType = backendBatchType
+
+ override def rowType(): Convention.RowType = backendRowType
}
-object VeloxColumnarToCarrierRowExec {
- def enforce(child: SparkPlan): SparkPlan = {
- Transitions.enforceReq(
- child,
- ConventionReq.ofRow(ConventionReq.RowType.Is(VeloxCarrierRowType)))
+object SharedColumnarToCarrierRowExec {
+ def enforce(child: SparkPlan, rowType: Convention.RowType): SparkPlan = {
+ Transitions.enforceReq(child, ConventionReq.ofRow(ConventionReq.RowType.Is(rowType)))
}
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala
index 79d350e90a..dc0514a60e 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxCarrierRowType.scala
@@ -16,11 +16,11 @@
*/
package org.apache.gluten.backendsapi.velox
-import org.apache.gluten.execution.VeloxColumnarToCarrierRowExec
-import org.apache.gluten.extension.columnar.transition.Convention
+import org.apache.gluten.execution.{SharedCarrierRowType, VeloxColumnarToCarrierRowExec}
+import org.apache.gluten.extension.columnar.transition.{Convention, Transition}
-object VeloxCarrierRowType extends Convention.RowType {
- override protected[this] def registerTransitions(): Unit = {
- fromBatch(VeloxBatchType, VeloxColumnarToCarrierRowExec.apply)
- }
+object VeloxCarrierRowType extends SharedCarrierRowType {
+ override protected def backendBatchType: Convention.BatchType = VeloxBatchType
+
+ override protected def toCarrierRowTransition: Transition = VeloxColumnarToCarrierRowExec.apply
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala b/backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala
index 65ce184861..b1038cb5bf 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/execution/VeloxColumnarToCarrierRowExec.scala
@@ -17,22 +17,20 @@
package org.apache.gluten.execution
import org.apache.gluten.backendsapi.velox.{VeloxBatchType, VeloxCarrierRowType}
-import org.apache.gluten.extension.columnar.transition.{Convention, ConventionReq, Transitions}
+import org.apache.gluten.extension.columnar.transition.Convention
import org.apache.spark.sql.execution.SparkPlan
case class VeloxColumnarToCarrierRowExec(override val child: SparkPlan)
- extends ColumnarToCarrierRowExecBase {
- override protected def fromBatchType(): Convention.BatchType = VeloxBatchType
- override def rowType(): Convention.RowType = VeloxCarrierRowType
+ extends SharedColumnarToCarrierRowExec(child) {
+ override protected def backendBatchType: Convention.BatchType = VeloxBatchType
+ override protected def backendRowType: Convention.RowType = VeloxCarrierRowType
override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =
copy(child = newChild)
}
object VeloxColumnarToCarrierRowExec {
def enforce(child: SparkPlan): SparkPlan = {
- Transitions.enforceReq(
- child,
- ConventionReq.ofRow(ConventionReq.RowType.Is(VeloxCarrierRowType)))
+ SharedColumnarToCarrierRowExec.enforce(child, VeloxCarrierRowType)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 18/38: [GLUTEN][CORE] Share Iceberg visitor and proto helpers
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 992ce0407f0b973986547fa7349aa4701b6aaa29
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:38:02 2026 +0800
[GLUTEN][CORE] Share Iceberg visitor and proto helpers
Move the backend-neutral Iceberg nested-field visitor, partition transform utility, and related protobuf schemas into backends-core so Iceberg write plumbing can be reused across backends without duplicating the same helper definitions.
---
backends-core/pom.xml | 45 ++++++++++++++++++++++
.../apache/gluten/IcebergNestedFieldVisitor.java | 0
.../apache/gluten/proto/IcebergNestedField.proto | 2 +-
.../apache/gluten/proto/IcebergPartitionSpec.proto | 2 +-
.../iceberg/transforms/IcebergTransformUtil.scala | 0
5 files changed, 47 insertions(+), 2 deletions(-)
diff --git a/backends-core/pom.xml b/backends-core/pom.xml
index 6e3fa13b70..113cbed416 100644
--- a/backends-core/pom.xml
+++ b/backends-core/pom.xml
@@ -92,4 +92,49 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>iceberg</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ <version>${protobuf.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-spark-runtime-${sparkbundle.version}_${scala.binary.version}</artifactId>
+ <version>${iceberg.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.xolstice.maven.plugins</groupId>
+ <artifactId>protobuf-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>compile-gluten-iceberg-proto</id>
+ <goals>
+ <goal>compile</goal>
+ </goals>
+ <phase>generate-sources</phase>
+ <configuration>
+ <protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
+ <protoSourceRoot>${project.basedir}/src-iceberg/main/resources/org/apache/gluten/proto</protoSourceRoot>
+ <clearOutputDirectory>false</clearOutputDirectory>
+ <checkStaleness>true</checkStaleness>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/backends-velox/src-iceberg/main/java/org/apache/gluten/IcebergNestedFieldVisitor.java b/backends-core/src-iceberg/main/java/org/apache/gluten/IcebergNestedFieldVisitor.java
similarity index 100%
rename from backends-velox/src-iceberg/main/java/org/apache/gluten/IcebergNestedFieldVisitor.java
rename to backends-core/src-iceberg/main/java/org/apache/gluten/IcebergNestedFieldVisitor.java
diff --git a/backends-velox/src/main/resources/org/apache/gluten/proto/IcebergNestedField.proto b/backends-core/src-iceberg/main/resources/org/apache/gluten/proto/IcebergNestedField.proto
similarity index 99%
rename from backends-velox/src/main/resources/org/apache/gluten/proto/IcebergNestedField.proto
rename to backends-core/src-iceberg/main/resources/org/apache/gluten/proto/IcebergNestedField.proto
index eb5639104b..5a6956fca4 100644
--- a/backends-velox/src/main/resources/org/apache/gluten/proto/IcebergNestedField.proto
+++ b/backends-core/src-iceberg/main/resources/org/apache/gluten/proto/IcebergNestedField.proto
@@ -9,4 +9,4 @@ option java_multiple_files = true;
message IcebergNestedField {
int32 id = 1;
repeated IcebergNestedField children = 2;
-}
\ No newline at end of file
+}
diff --git a/backends-velox/src/main/resources/org/apache/gluten/proto/IcebergPartitionSpec.proto b/backends-core/src-iceberg/main/resources/org/apache/gluten/proto/IcebergPartitionSpec.proto
similarity index 99%
rename from backends-velox/src/main/resources/org/apache/gluten/proto/IcebergPartitionSpec.proto
rename to backends-core/src-iceberg/main/resources/org/apache/gluten/proto/IcebergPartitionSpec.proto
index e59fb88a9c..4718530199 100644
--- a/backends-velox/src/main/resources/org/apache/gluten/proto/IcebergPartitionSpec.proto
+++ b/backends-core/src-iceberg/main/resources/org/apache/gluten/proto/IcebergPartitionSpec.proto
@@ -26,4 +26,4 @@ message IcebergPartitionField {
message IcebergPartitionSpec {
int32 spec_id = 1; // Field name uses snake_case per protobuf conventions
repeated IcebergPartitionField fields = 2;
-}
\ No newline at end of file
+}
diff --git a/backends-velox/src-iceberg/main/scala/org/apache/iceberg/transforms/IcebergTransformUtil.scala b/backends-core/src-iceberg/main/scala/org/apache/iceberg/transforms/IcebergTransformUtil.scala
similarity index 100%
rename from backends-velox/src-iceberg/main/scala/org/apache/iceberg/transforms/IcebergTransformUtil.scala
rename to backends-core/src-iceberg/main/scala/org/apache/iceberg/transforms/IcebergTransformUtil.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 12/38: [GLUTEN][CORE] Expose shuffle reader metrics iterator delegate
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 1ae8a16664ed000e4baa5ed2db283c5fcedb01ba
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 21:18:02 2026 +0800
[GLUTEN][CORE] Expose shuffle reader metrics iterator delegate
---
.../sql/execution/ShuffledColumnarBatchRDD.scala | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ShuffledColumnarBatchRDD.scala b/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ShuffledColumnarBatchRDD.scala
index 0642c3a247..fde17aaeed 100644
--- a/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ShuffledColumnarBatchRDD.scala
+++ b/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ShuffledColumnarBatchRDD.scala
@@ -28,6 +28,19 @@ import org.apache.spark.sql.vectorized.ColumnarBatch
final private case class ShuffledColumnarBatchRDDPartition(index: Int, spec: ShufflePartitionSpec)
extends Partition
+/** Wrap metrics with named class, so callers can access the delegate iterator when needed. */
+class ShuffleReaderWithMetricsIterator(
+ val delegate: Iterator[Product2[Int, ColumnarBatch]],
+ sqlMetricsReporter: SQLColumnarShuffleReadMetricsReporter)
+ extends Iterator[ColumnarBatch] {
+ override def hasNext: Boolean = delegate.hasNext
+ override def next(): ColumnarBatch = {
+ val batch = delegate.next()._2
+ sqlMetricsReporter.incBatchesRecordsRead(batch.numRows())
+ batch
+ }
+}
+
/** [[ShuffledColumnarBatchRDD]] is the columnar version of [[org.apache.spark.rdd.ShuffledRDD]]. */
class ShuffledColumnarBatchRDD(
var dependency: ShuffleDependency[Int, ColumnarBatch, ColumnarBatch],
@@ -132,11 +145,9 @@ class ShuffledColumnarBatchRDD(
context,
sqlMetricsReporter)
}
- reader.read().asInstanceOf[Iterator[Product2[Int, ColumnarBatch]]].map {
- case (_, batch: ColumnarBatch) =>
- sqlMetricsReporter.incBatchesRecordsRead(batch.numRows())
- batch
- }
+ new ShuffleReaderWithMetricsIterator(
+ reader.read().asInstanceOf[Iterator[Product2[Int, ColumnarBatch]]],
+ sqlMetricsReporter)
}
override def clearDependencies(): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 11/38: [GLUTEN][CORE] Add backend hook for sequence expressions
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit b59a38602fc54841a86074847e7e18449251894e
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 21:18:02 2026 +0800
[GLUTEN][CORE] Add backend hook for sequence expressions
---
.../scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala | 8 ++++++++
.../scala/org/apache/gluten/expression/ExpressionConverter.scala | 5 +++++
2 files changed, 13 insertions(+)
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
index ee347c183e..6aa2f7f974 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
@@ -286,6 +286,14 @@ trait SparkPlanExecApi {
throw new GlutenNotSupportException("map_entries is not supported")
}
+ /** Transform sequence to Substrait. */
+ def genSequenceTransformer(
+ substraitExprName: String,
+ children: Seq[ExpressionTransformer],
+ expr: Sequence): ExpressionTransformer = {
+ GenericExpressionTransformer(substraitExprName, children, expr)
+ }
+
/** Transform array filter to Substrait. */
def genArrayFilterTransformer(
substraitExprName: String,
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala b/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala
index 7969d30502..bb52801e8e 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala
@@ -340,6 +340,11 @@ object ExpressionConverter extends SQLConfHelper with Logging {
substraitExprName,
replaceWithExpressionTransformer0(m.child, attributeSeq, expressionsMap),
m)
+ case s: Sequence =>
+ BackendsApiManager.getSparkPlanExecApiInstance.genSequenceTransformer(
+ substraitExprName,
+ s.children.map(replaceWithExpressionTransformer0(_, attributeSeq, expressionsMap)),
+ s)
case e: Explode =>
ExplodeTransformer(
substraitExprName,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 03/38: [GLUTEN][CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit ab54bf2a8167b8ebbd015d06069244d8af9559d8
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Fri Jun 26 17:19:40 2026 +0800
[GLUTEN][CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers
Introduce an opt-in framework to pass stage input statistics (scan / shuffle /
broadcast) to the native engine as estimated row-size hints.
- Add InputStats / InputStatsKind / ShuffleStageWrapper and ApplyStageInputStatsRule.
- Add RelNode.childNode() with implementations across all rel nodes; carry
rowSize / inputStats on ReadRelNode and InputIteratorRelNode; expose
PlanNode.getRelNodes() and AdvancedExtensionNode optimization/enhancement getters.
- Add BasicScanExecTransformer.getInputStats hook and optional inputStats field on
FileSourceScanExecTransformer; thread wsContext through genFirstStageIterator and
the whole-stage RDDs; inject shuffle/broadcast stats in InputIteratorTransformer.
- Gate everything behind spark.gluten.sql.enablePassStageInputStats (default false),
so velox / clickhouse behavior is unchanged.
Generated-by: TraeCli openrouter-3o
---
.../backendsapi/clickhouse/CHIteratorApi.scala | 3 +-
.../backendsapi/velox/VeloxIteratorApi.scala | 3 +-
.../execution/adaptive/ShuffleStageWrapper.scala | 83 +++++++++++++
.../extensions/AdvancedExtensionNode.java | 8 ++
.../org/apache/gluten/substrait/plan/PlanNode.java | 4 +
.../gluten/substrait/rel/AggregateRelNode.java | 6 +
.../apache/gluten/substrait/rel/CrossRelNode.java | 10 ++
.../apache/gluten/substrait/rel/ExpandRelNode.java | 6 +
.../apache/gluten/substrait/rel/FetchRelNode.java | 7 ++
.../apache/gluten/substrait/rel/FilterRelNode.java | 7 ++
.../gluten/substrait/rel/GenerateRelNode.java | 6 +
.../gluten/substrait/rel/InputIteratorRelNode.java | 40 +++++++
.../apache/gluten/substrait/rel/JoinRelNode.java | 10 ++
.../gluten/substrait/rel/ProjectRelNode.java | 6 +
.../apache/gluten/substrait/rel/ReadRelNode.java | 57 ++++++++-
.../org/apache/gluten/substrait/rel/RelNode.java | 3 +
.../apache/gluten/substrait/rel/SetRelNode.java | 5 +
.../apache/gluten/substrait/rel/SortRelNode.java | 6 +
.../org/apache/gluten/substrait/rel/TopNNode.java | 6 +
.../substrait/rel/WindowGroupLimitRelNode.java | 6 +
.../apache/gluten/substrait/rel/WindowRelNode.java | 6 +
.../apache/gluten/substrait/rel/WriteRelNode.java | 6 +
.../apache/gluten/backendsapi/IteratorApi.scala | 3 +-
.../org/apache/gluten/config/GlutenConfig.scala | 11 ++
.../execution/BasicScanExecTransformer.scala | 10 +-
.../execution/FileSourceScanExecTransformer.scala | 11 +-
.../execution/GlutenWholeStageColumnarRDD.scala | 14 ++-
.../gluten/execution/WholeStageTransformer.scala | 6 +-
.../execution/WholeStageZippedPartitionsRDD.scala | 11 +-
.../extension/ApplyStageInputStatsRule.scala | 132 +++++++++++++++++++++
.../ColumnarCollapseTransformStages.scala | 35 +++++-
31 files changed, 513 insertions(+), 14 deletions(-)
diff --git a/backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHIteratorApi.scala b/backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHIteratorApi.scala
index f9820178c2..8e20a72a65 100644
--- a/backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHIteratorApi.scala
+++ b/backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHIteratorApi.scala
@@ -277,7 +277,8 @@ class CHIteratorApi extends IteratorApi with Logging with LogLevelUtil {
updateNativeMetrics: IMetrics => Unit,
partitionIndex: Int,
inputIterators: Seq[Iterator[ColumnarBatch]] = Seq(),
- enableCudf: Boolean = false
+ enableCudf: Boolean = false,
+ wsContext: WholeStageTransformContext = null
): Iterator[ColumnarBatch] = {
require(
diff --git a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala
index 73867aace9..9ad1e8dc8e 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala
@@ -190,7 +190,8 @@ class VeloxIteratorApi extends IteratorApi with Logging {
updateNativeMetrics: IMetrics => Unit,
partitionIndex: Int,
inputIterators: Seq[Iterator[ColumnarBatch]] = Seq(),
- enableCudf: Boolean = false): Iterator[ColumnarBatch] = {
+ enableCudf: Boolean = false,
+ wsContext: WholeStageTransformContext = null): Iterator[ColumnarBatch] = {
assert(
inputPartition.isInstanceOf[GlutenPartition],
"Velox backend only accept GlutenPartition.")
diff --git a/gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShuffleStageWrapper.scala b/gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShuffleStageWrapper.scala
new file mode 100644
index 0000000000..47c23dc9c7
--- /dev/null
+++ b/gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShuffleStageWrapper.scala
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.MapOutputStatistics
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.trees.TreeNodeTag
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.exchange.ENSURE_REQUIREMENTS
+
+/**
+ * Represents stage input stats, eg scan or shuffle
+ *
+ * @param known
+ * @param sizeInBytes
+ * @param rowCount
+ * @param bytesByPartitionId
+ */
+sealed trait InputStatsKind
+case object ScanInputStats extends InputStatsKind
+case object BroadcastStats extends InputStatsKind
+case object ShuffleStats extends InputStatsKind
+case object UnknownStats extends InputStatsKind
+case class InputStats(
+ known: Boolean,
+ sizeInBytes: BigInt,
+ rowCount: BigInt,
+ bytesByPartitionId: Array[Long],
+ inputStatsKind: InputStatsKind)
+ extends Serializable {
+ override def toString: String = {
+ s"known : ($known) sizeInBytes:($sizeInBytes), rowCount: ($rowCount), " +
+ s"bytesByPartitionId: [${bytesByPartitionId.mkString(",")}], statsKind: $inputStatsKind"
+ }
+}
+object ShuffleStageWrapper extends Logging {
+ val INPUT_STATS: TreeNodeTag[java.util.List[InputStats]] = TreeNodeTag("InputStats")
+ def unapply(plan: SparkPlan): Option[ShuffleQueryStageRuntimeStats] = plan match {
+ case s: ShuffleQueryStageExec
+ if s.isMaterialized && s.mapStats.isDefined &&
+ s.shuffle.shuffleOrigin == ENSURE_REQUIREMENTS =>
+ logInfo("hit InputIteratorTransformer ShuffleQueryStageExec with stats")
+ Some(
+ ShuffleQueryStageRuntimeStats(
+ s.getRuntimeStatistics.sizeInBytes,
+ s.getRuntimeStatistics.rowCount,
+ s.mapStats))
+ case a: AQEShuffleReadExec if a.child.isInstanceOf[ShuffleQueryStageExec] =>
+ logInfo("hit InputIteratorTransformer AQEShuffleReadExec with stats")
+ val queryStageExec = a.child.asInstanceOf[ShuffleQueryStageExec]
+ Some(
+ ShuffleQueryStageRuntimeStats(
+ queryStageExec.getRuntimeStatistics.sizeInBytes,
+ queryStageExec.getRuntimeStatistics.rowCount,
+ queryStageExec.mapStats))
+ case b: BroadcastQueryStageExec if b.isMaterialized =>
+ logInfo("hit InputIteratorTransformer BroadcastQueryStageExec with stats")
+ Some(
+ ShuffleQueryStageRuntimeStats(
+ b.getRuntimeStatistics.sizeInBytes,
+ b.getRuntimeStatistics.rowCount,
+ None))
+ case _ => None
+ }
+}
+case class ShuffleQueryStageRuntimeStats(
+ sizeInBytes: BigInt,
+ rowCount: Option[BigInt] = None,
+ mapOutputStatistics: Option[MapOutputStatistics] = None)
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/extensions/AdvancedExtensionNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/extensions/AdvancedExtensionNode.java
index f313888d4f..b75f4e587f 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/extensions/AdvancedExtensionNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/extensions/AdvancedExtensionNode.java
@@ -40,6 +40,14 @@ public class AdvancedExtensionNode implements Serializable {
this.enhancement = enhancement;
}
+ public Any getOptimization() {
+ return optimization;
+ }
+
+ public Any getEnhancement() {
+ return enhancement;
+ }
+
public AdvancedExtension toProtobuf() {
AdvancedExtension.Builder extensionBuilder = AdvancedExtension.newBuilder();
if (optimization != null) {
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/plan/PlanNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/plan/PlanNode.java
index 0dd4a74bbc..373508392a 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/plan/PlanNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/plan/PlanNode.java
@@ -77,4 +77,8 @@ public class PlanNode implements Serializable {
}
return planBuilder.build();
}
+
+ public List<RelNode> getRelNodes() {
+ return relNodes;
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/AggregateRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/AggregateRelNode.java
index 61ed751e45..075aa0fbe4 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/AggregateRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/AggregateRelNode.java
@@ -26,6 +26,7 @@ import io.substrait.proto.RelCommon;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class AggregateRelNode implements RelNode, Serializable {
@@ -83,4 +84,9 @@ public class AggregateRelNode implements RelNode, Serializable {
builder.setAggregate(aggBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/CrossRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/CrossRelNode.java
index 338a10b4d6..e2faa88103 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/CrossRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/CrossRelNode.java
@@ -24,6 +24,8 @@ import io.substrait.proto.Rel;
import io.substrait.proto.RelCommon;
import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
public class CrossRelNode implements RelNode, Serializable {
private final RelNode left;
@@ -69,4 +71,12 @@ public class CrossRelNode implements RelNode, Serializable {
}
return Rel.newBuilder().setCross(crossRelBuilder.build()).build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ List<RelNode> children = new ArrayList<>();
+ children.add(left);
+ children.add(right);
+ return children;
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ExpandRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ExpandRelNode.java
index 25d007e564..844c21910c 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ExpandRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ExpandRelNode.java
@@ -25,6 +25,7 @@ import io.substrait.proto.RelCommon;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class ExpandRelNode implements RelNode, Serializable {
@@ -76,4 +77,9 @@ public class ExpandRelNode implements RelNode, Serializable {
builder.setExpand(expandBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FetchRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FetchRelNode.java
index 3c8b991c5b..7d8d591172 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FetchRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FetchRelNode.java
@@ -23,6 +23,8 @@ import io.substrait.proto.Rel;
import io.substrait.proto.RelCommon;
import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
public class FetchRelNode implements RelNode, Serializable {
private final RelNode input;
@@ -66,4 +68,9 @@ public class FetchRelNode implements RelNode, Serializable {
relBuilder.setFetch(fetchRelBuilder.build());
return relBuilder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FilterRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FilterRelNode.java
index 356b12b292..7a7d2fa366 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FilterRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FilterRelNode.java
@@ -24,6 +24,8 @@ import io.substrait.proto.Rel;
import io.substrait.proto.RelCommon;
import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
public class FilterRelNode implements RelNode, Serializable {
private final RelNode input;
@@ -60,4 +62,9 @@ public class FilterRelNode implements RelNode, Serializable {
builder.setFilter(filterBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/GenerateRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/GenerateRelNode.java
index efd4708919..1d1a7a58b5 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/GenerateRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/GenerateRelNode.java
@@ -24,6 +24,7 @@ import io.substrait.proto.Rel;
import io.substrait.proto.RelCommon;
import java.io.Serializable;
+import java.util.Collections;
import java.util.List;
public class GenerateRelNode implements RelNode, Serializable {
@@ -81,4 +82,9 @@ public class GenerateRelNode implements RelNode, Serializable {
relBuilder.setGenerate(generateRelBuilder.build());
return relBuilder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/InputIteratorRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/InputIteratorRelNode.java
index 55b0e313be..cf896c8a2f 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/InputIteratorRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/InputIteratorRelNode.java
@@ -17,12 +17,20 @@
package org.apache.gluten.substrait.rel;
import org.apache.gluten.expression.ConverterUtils;
+import org.apache.gluten.substrait.extensions.AdvancedExtensionNode;
+import org.apache.gluten.substrait.extensions.ExtensionBuilder;
import org.apache.gluten.substrait.type.TypeNode;
+import com.google.protobuf.Any;
+import com.google.protobuf.StringValue;
import io.substrait.proto.*;
+import org.apache.spark.sql.execution.adaptive.InputStats;
+import java.util.ArrayList;
import java.util.List;
+import scala.math.BigInt;
+
/**
* The relation for input iterator, e.g., the input is ColumnarShuffleExchange or RowToColumnarExec.
* It uses `ReadRel` as the substrait rel type.
@@ -32,6 +40,9 @@ public class InputIteratorRelNode implements RelNode {
private final List<TypeNode> types;
private final List<String> names;
private final Long iteratorIndex;
+ private BigInt rowSize;
+
+ private InputStats inputStats;
public InputIteratorRelNode(List<TypeNode> types, List<String> names, Long iteratorIndex) {
this.types = types;
@@ -39,6 +50,22 @@ public class InputIteratorRelNode implements RelNode {
this.iteratorIndex = iteratorIndex;
}
+ public BigInt getRowSize() {
+ return rowSize;
+ }
+
+ public void setRowSize(BigInt rowSize) {
+ this.rowSize = rowSize;
+ }
+
+ public InputStats getInputStats() {
+ return inputStats;
+ }
+
+ public void setInputStats(InputStats inputStats) {
+ this.inputStats = inputStats;
+ }
+
@Override
public Rel toProtobuf() {
Type.Struct.Builder structBuilder = Type.Struct.newBuilder();
@@ -59,8 +86,21 @@ public class InputIteratorRelNode implements RelNode {
LocalFilesBuilder.makeLocalFiles(ConverterUtils.ITERATOR_PREFIX() + iteratorIndex);
readBuilder.setLocalFiles(iteratorIndexNode.toProtobuf());
+ if (null != rowSize) {
+ Any inputRowSize =
+ Any.pack(StringValue.newBuilder().setValue("rowSize=" + rowSize.toLong() + "\n").build());
+ AdvancedExtensionNode advancedExtension =
+ ExtensionBuilder.makeAdvancedExtension(inputRowSize, null);
+ readBuilder.setAdvancedExtension(advancedExtension.toProtobuf());
+ }
+
Rel.Builder builder = Rel.newBuilder();
builder.setRead(readBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return new ArrayList<>();
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/JoinRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/JoinRelNode.java
index 2bd98500fe..fcf7a3fea3 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/JoinRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/JoinRelNode.java
@@ -24,6 +24,8 @@ import io.substrait.proto.Rel;
import io.substrait.proto.RelCommon;
import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
public class JoinRelNode implements RelNode, Serializable {
private final RelNode left;
@@ -79,4 +81,12 @@ public class JoinRelNode implements RelNode, Serializable {
return Rel.newBuilder().setJoin(joinBuilder.build()).build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ List<RelNode> children = new ArrayList<>();
+ children.add(left);
+ children.add(right);
+ return children;
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ProjectRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ProjectRelNode.java
index d1cccd1595..b5e486baaa 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ProjectRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ProjectRelNode.java
@@ -25,6 +25,7 @@ import io.substrait.proto.RelCommon;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class ProjectRelNode implements RelNode, Serializable {
@@ -78,4 +79,9 @@ public class ProjectRelNode implements RelNode, Serializable {
builder.setProject(projectBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java
index b82e05fd36..b44a9631f8 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java
@@ -18,19 +18,26 @@ package org.apache.gluten.substrait.rel;
import org.apache.gluten.substrait.expression.ExpressionNode;
import org.apache.gluten.substrait.extensions.AdvancedExtensionNode;
+import org.apache.gluten.substrait.extensions.ExtensionBuilder;
import org.apache.gluten.substrait.type.ColumnTypeNode;
import org.apache.gluten.substrait.type.TypeNode;
import org.apache.gluten.utils.SubstraitUtil;
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.StringValue;
import io.substrait.proto.NamedStruct;
import io.substrait.proto.ReadRel;
import io.substrait.proto.Rel;
import io.substrait.proto.RelCommon;
+import org.apache.spark.sql.execution.adaptive.InputStats;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
+import scala.math.BigInt;
+
public class ReadRelNode implements RelNode, Serializable {
private final List<TypeNode> types = new ArrayList<>();
private final List<String> names = new ArrayList<>();
@@ -39,6 +46,9 @@ public class ReadRelNode implements RelNode, Serializable {
private final AdvancedExtensionNode extensionNode;
private boolean streamKafka = false;
+ private BigInt rowSize;
+ private InputStats inputStats;
+
ReadRelNode(
List<TypeNode> types,
List<String> names,
@@ -52,6 +62,22 @@ public class ReadRelNode implements RelNode, Serializable {
this.extensionNode = extensionNode;
}
+ public BigInt getRowSize() {
+ return rowSize;
+ }
+
+ public void setRowSize(BigInt rowSize) {
+ this.rowSize = rowSize;
+ }
+
+ public InputStats getInputStats() {
+ return inputStats;
+ }
+
+ public void setInputStats(InputStats inputStats) {
+ this.inputStats = inputStats;
+ }
+
public void setStreamKafka(boolean streamKafka) {
this.streamKafka = streamKafka;
}
@@ -74,11 +100,40 @@ public class ReadRelNode implements RelNode, Serializable {
}
if (extensionNode != null) {
- readBuilder.setAdvancedExtension(extensionNode.toProtobuf());
+ if (null != rowSize) {
+ Any optimization = extensionNode.getOptimization();
+ Any enhancement = extensionNode.getEnhancement();
+ try {
+ Any any = Any.parseFrom(optimization.toByteArray());
+ String newOptimizationStr = any.getValue() + "rowSize=" + rowSize.toLong() + "\n";
+ Any newOptimization =
+ Any.pack(StringValue.newBuilder().setValue(newOptimizationStr).build());
+ readBuilder.setAdvancedExtension(
+ new AdvancedExtensionNode(newOptimization, enhancement).toProtobuf());
+ } catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
+ }
+ } else {
+ readBuilder.setAdvancedExtension(extensionNode.toProtobuf());
+ }
+ } else {
+ if (null != rowSize) {
+ Any inputRowSize =
+ Any.pack(
+ StringValue.newBuilder().setValue("rowSize=" + rowSize.toLong() + "\n").build());
+ AdvancedExtensionNode advancedExtension =
+ ExtensionBuilder.makeAdvancedExtension(inputRowSize, null);
+ readBuilder.setAdvancedExtension(advancedExtension.toProtobuf());
+ }
}
Rel.Builder builder = Rel.newBuilder();
builder.setRead(readBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return new ArrayList<>();
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelNode.java
index 3dacd947c0..817e39cf9f 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelNode.java
@@ -19,6 +19,7 @@ package org.apache.gluten.substrait.rel;
import io.substrait.proto.Rel;
import java.io.Serializable;
+import java.util.List;
/** Contains helper functions for constructing substrait relations. */
public interface RelNode extends Serializable {
@@ -28,4 +29,6 @@ public interface RelNode extends Serializable {
* @return A rel protobuf
*/
Rel toProtobuf();
+
+ List<RelNode> childNode();
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SetRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SetRelNode.java
index ddcfb1701d..eab234acb5 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SetRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SetRelNode.java
@@ -59,4 +59,9 @@ public class SetRelNode implements RelNode, Serializable {
builder.setSet(setBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return inputs;
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SortRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SortRelNode.java
index ec7368cd3a..5266a4bd8e 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SortRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SortRelNode.java
@@ -24,6 +24,7 @@ import io.substrait.proto.SortField;
import io.substrait.proto.SortRel;
import java.io.Serializable;
+import java.util.Collections;
import java.util.List;
public class SortRelNode implements RelNode, Serializable {
@@ -67,4 +68,9 @@ public class SortRelNode implements RelNode, Serializable {
builder.setSort(sortBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/TopNNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/TopNNode.java
index a33d7aaca5..efd4f09a9a 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/TopNNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/TopNNode.java
@@ -25,6 +25,7 @@ import io.substrait.proto.TopNRel;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class TopNNode implements RelNode, Serializable {
@@ -73,4 +74,9 @@ public class TopNNode implements RelNode, Serializable {
relBuilder.setTopN(topNBuilder.build());
return relBuilder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowGroupLimitRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowGroupLimitRelNode.java
index a7a0ea62aa..e4a6e8b647 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowGroupLimitRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowGroupLimitRelNode.java
@@ -26,6 +26,7 @@ import io.substrait.proto.WindowGroupLimitRel;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class WindowGroupLimitRelNode implements RelNode, Serializable {
@@ -88,4 +89,9 @@ public class WindowGroupLimitRelNode implements RelNode, Serializable {
builder.setWindowGroupLimit(windowBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowRelNode.java
index 415bbc350f..409d143bf4 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowRelNode.java
@@ -27,6 +27,7 @@ import io.substrait.proto.WindowRel;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class WindowRelNode implements RelNode, Serializable {
@@ -93,4 +94,9 @@ public class WindowRelNode implements RelNode, Serializable {
builder.setWindow(windowBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WriteRelNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WriteRelNode.java
index 74ffc8282c..af6aba7f1e 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WriteRelNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WriteRelNode.java
@@ -25,6 +25,7 @@ import io.substrait.proto.*;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
public class WriteRelNode implements RelNode, Serializable {
@@ -83,4 +84,9 @@ public class WriteRelNode implements RelNode, Serializable {
builder.setWrite(writeBuilder.build());
return builder.build();
}
+
+ @Override
+ public List<RelNode> childNode() {
+ return Collections.singletonList(input);
+ }
}
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala
index a6b0053593..2cb45b97a8 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala
@@ -68,7 +68,8 @@ trait IteratorApi {
updateNativeMetrics: IMetrics => Unit,
partitionIndex: Int,
inputIterators: Seq[Iterator[ColumnarBatch]] = Seq(),
- enableCudf: Boolean = false
+ enableCudf: Boolean = false,
+ wsContext: WholeStageTransformContext = null
): Iterator[ColumnarBatch]
/**
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala b/gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala
index 053d715125..d6e8fded5f 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala
@@ -325,6 +325,8 @@ class GlutenConfig(conf: SQLConf) extends GlutenCoreConfig(conf) {
def enableFallbackReport: Boolean = getConf(FALLBACK_REPORTER_ENABLED)
+ def enablePassStageInputStats: Boolean = getConf(GLUTEN_PASS_STAGE_INPUT_STATS_ENABLED)
+
def failOnFallback: Boolean = getConf(FALLBACK_FAIL_ON_FALLBACK)
def debug: Boolean = getConf(DEBUG_ENABLED)
@@ -1431,6 +1433,15 @@ object GlutenConfig extends ConfigRegistry {
.booleanConf
.createWithDefault(true)
+ val GLUTEN_PASS_STAGE_INPUT_STATS_ENABLED =
+ buildConf("spark.gluten.sql.enablePassStageInputStats")
+ .internal()
+ .doc("When true, pass stage input stats (scan/shuffle/broadcast) to the native engine " +
+ "as estimated row size hints. Disabled by default and no-op for backends that do not " +
+ "consume the hints.")
+ .booleanConf
+ .createWithDefault(false)
+
val FALLBACK_FAIL_ON_FALLBACK =
buildConf("spark.gluten.sql.columnar.failOnFallback")
.internal()
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala
index 3b142fecb9..6d265100d0 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala
@@ -22,11 +22,12 @@ import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
import org.apache.gluten.substrait.`type`.ColumnTypeNode
import org.apache.gluten.substrait.SubstraitContext
import org.apache.gluten.substrait.extensions.ExtensionBuilder
-import org.apache.gluten.substrait.rel.{RelBuilder, SplitInfo}
+import org.apache.gluten.substrait.rel.{ReadRelNode, RelBuilder, SplitInfo}
import org.apache.gluten.substrait.rel.LocalFilesNode.ReadFileFormat
import org.apache.spark.Partition
import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.execution.adaptive.InputStats
import com.google.protobuf.StringValue
import io.substrait.proto.NamedStruct
@@ -94,6 +95,8 @@ trait BasicScanExecTransformer extends LeafTransformSupport with BaseDataSource
/** Returns the file format properties. */
def getProperties: Map[String, String] = Map.empty
+ def getInputStats: Option[InputStats] = Option.empty
+
override def getSplitInfos: Seq[SplitInfo] = {
getSplitInfosFromPartitions(getPartitionWithReadFileFormats)
}
@@ -183,6 +186,11 @@ trait BasicScanExecTransformer extends LeafTransformSupport with BaseDataSource
extensionNode,
context,
context.nextOperatorId(this.nodeName))
+ getInputStats.foreach(
+ inputStats => {
+ logInfo(s"hit scan inputStats with $inputStats")
+ readNode.asInstanceOf[ReadRelNode].setInputStats(inputStats)
+ })
TransformContext(output, readNode)
}
}
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/FileSourceScanExecTransformer.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/FileSourceScanExecTransformer.scala
index 8fea175d7e..83023f027a 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/FileSourceScanExecTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/FileSourceScanExecTransformer.scala
@@ -29,6 +29,7 @@ import org.apache.spark.sql.catalyst.plans.QueryPlan
import org.apache.spark.sql.catalyst.util.truncatedString
import org.apache.spark.sql.connector.read.streaming.SparkDataStream
import org.apache.spark.sql.execution.FileSourceScanExecShim
+import org.apache.spark.sql.execution.adaptive.InputStats
import org.apache.spark.sql.execution.datasources.HadoopFsRelation
import org.apache.spark.sql.execution.metric.SQLMetric
import org.apache.spark.sql.types.StructType
@@ -48,7 +49,8 @@ case class FileSourceScanExecTransformer(
override val dataFilters: Seq[Expression],
override val tableIdentifier: Option[TableIdentifier],
override val disableBucketedScan: Boolean = false,
- override val pushDownFilters: Option[Seq[Expression]] = None)
+ override val pushDownFilters: Option[Seq[Expression]] = None,
+ inputStats: Option[InputStats] = None)
extends FileSourceScanExecTransformerBase(
relation,
stream,
@@ -62,6 +64,10 @@ case class FileSourceScanExecTransformer(
disableBucketedScan
) {
+ override def getInputStats: Option[InputStats] = {
+ inputStats
+ }
+
override def doCanonicalize(): FileSourceScanExecTransformer = {
FileSourceScanExecTransformer(
relation,
@@ -78,7 +84,8 @@ case class FileSourceScanExecTransformer(
QueryPlan.normalizePredicates(dataFilters, output),
None,
disableBucketedScan,
- pushDownFilters.map(QueryPlan.normalizePredicates(_, output))
+ pushDownFilters.map(QueryPlan.normalizePredicates(_, output)),
+ inputStats
)
}
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala
index 4a00dbb587..afec9cc10f 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala
@@ -17,6 +17,8 @@
package org.apache.gluten.execution
import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.extension.ApplyStageInputStatsRule
import org.apache.gluten.metrics.{GlutenTimeMetric, IMetrics}
import org.apache.gluten.substrait.rel.SplitInfo
@@ -58,12 +60,19 @@ class GlutenWholeStageColumnarRDD(
pipelineTime: SQLMetric,
updateInputMetrics: InputMetricsWrapper => Unit,
updateNativeMetrics: IMetrics => Unit,
- enableCudf: Boolean = false)
+ enableCudf: Boolean = false,
+ wsContext: WholeStageTransformContext = null)
extends RDD[ColumnarBatch](sc, rdds.getDependencies) {
override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = {
GlutenTimeMetric.millis(pipelineTime) {
_ =>
+ if (GlutenConfig.get.enablePassStageInputStats && wsContext != null) {
+ ApplyStageInputStatsRule.setStageInputStatsToInputNode(
+ wsContext,
+ split.index,
+ rdds.getPartitionLength)
+ }
val (inputPartition, inputColumnarRDDPartitions) = castNativePartition(split)
val inputIterators = rdds.getIterators(inputColumnarRDDPartitions, context)
BackendsApiManager.getIteratorApiInstance.genFirstStageIterator(
@@ -74,7 +83,8 @@ class GlutenWholeStageColumnarRDD(
updateNativeMetrics,
split.index,
inputIterators,
- enableCudf
+ enableCudf,
+ wsContext
)
}
}
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
index 8de9d407bf..1f3ae4d753 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
@@ -379,7 +379,8 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
wsCtx.substraitContext.registeredJoinParams,
wsCtx.substraitContext.registeredAggregationParams
),
- wsCtx.enableCudf
+ wsCtx.enableCudf,
+ wsCtx
)
val allInputPartitions = leafTransformers.map(_.getPartitions)
@@ -434,7 +435,8 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
wsCtx.substraitContext.registeredJoinParams,
wsCtx.substraitContext.registeredAggregationParams
),
- materializeInput
+ materializeInput,
+ inputRDDs.getPartitionLength
)
}
}
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala
index 393716bb7b..73068241f7 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala
@@ -17,6 +17,8 @@
package org.apache.gluten.execution
import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.extension.ApplyStageInputStatsRule
import org.apache.gluten.metrics.{GlutenTimeMetric, IMetrics}
import org.apache.spark.{Partition, SparkConf, SparkContext, TaskContext}
@@ -36,12 +38,19 @@ class WholeStageZippedPartitionsRDD(
resCtx: WholeStageTransformContext,
pipelineTime: SQLMetric,
updateNativeMetrics: IMetrics => Unit,
- materializeInput: Boolean)
+ materializeInput: Boolean,
+ partitionLength: Int = 0)
extends RDD[ColumnarBatch](sc, rdds.getDependencies) {
override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = {
GlutenTimeMetric.millis(pipelineTime) {
_ =>
+ if (GlutenConfig.get.enablePassStageInputStats) {
+ ApplyStageInputStatsRule.setStageInputStatsToInputNode(
+ resCtx,
+ split.index,
+ partitionLength)
+ }
val partitions = split.asInstanceOf[ZippedPartitionsPartition].inputColumnarRDDPartitions
val inputIterators: Seq[Iterator[ColumnarBatch]] = rdds.getIterators(partitions, context)
BackendsApiManager.getIteratorApiInstance
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/extension/ApplyStageInputStatsRule.scala b/gluten-substrait/src/main/scala/org/apache/gluten/extension/ApplyStageInputStatsRule.scala
new file mode 100644
index 0000000000..bb91c3ae9b
--- /dev/null
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/extension/ApplyStageInputStatsRule.scala
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension
+
+import org.apache.gluten.execution.WholeStageTransformContext
+import org.apache.gluten.substrait.rel.{InputIteratorRelNode, ReadRelNode, RelNode}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.{CoalescedMapperPartitionSpec, CoalescedPartitionSpec, PartialMapperPartitionSpec, PartialReducerPartitionSpec, ShufflePartitionSpec}
+import org.apache.spark.sql.execution.adaptive.{BroadcastStats, InputStats, ScanInputStats, ShuffleStats}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+
+import scala.collection.{mutable, Seq}
+
+object ApplyStageInputStatsRule extends Logging {
+ def setStageInputStatsToInputNode(
+ ws: WholeStageTransformContext,
+ index: Int,
+ partitionLength: Int): Unit = {
+ val relNodesQueue: mutable.Queue[RelNode] = mutable.Queue()
+ val nodes = ws.root.getRelNodes
+ if (nodes != null) {
+ nodes.forEach(relNodesQueue.enqueue(_))
+ }
+ while (relNodesQueue.nonEmpty) {
+ val node = relNodesQueue.dequeue()
+ node match {
+ case input: InputIteratorRelNode =>
+ val stats = input.getInputStats
+ if (null != stats) {
+ val totalBytes = Math.max(stats.bytesByPartitionId.sum, 1)
+ val estimatedRowCount =
+ if (stats.inputStatsKind == BroadcastStats) {
+ stats.rowCount.toLong
+ } else {
+ (1.0d * stats.rowCount.toLong / totalBytes * stats.bytesByPartitionId(index)).toLong
+ }
+ logInfo(s"pass input stats idx: $index with estimated row count $estimatedRowCount")
+ input.setRowSize(estimatedRowCount)
+ }
+ case scan: ReadRelNode =>
+ val stats = scan.getInputStats
+ if (null != stats) {
+ val estimatedRowCount = (1.0d * stats.rowCount.toLong / partitionLength).toLong
+ logInfo(s"pass scan stats idx: $index with estimated row count $estimatedRowCount")
+ scan.setRowSize(estimatedRowCount)
+ }
+ case _ =>
+ }
+ node.childNode().forEach(relNodesQueue.enqueue(_))
+ }
+ }
+
+ /** Compute task stats for each partition */
+ def recomputeInputStatsForAQEShuffleReadExec(
+ original: InputStats,
+ aqeShufflePartitions: Seq[ShufflePartitionSpec]): InputStats = {
+ val originalBytesByPartition = original.bytesByPartitionId
+ val totalBytes = originalBytesByPartition.sum
+ val rowCount = original.rowCount
+ val aqeReadPartitionsStats = new mutable.ArrayBuffer[Long]()
+ aqeShufflePartitions.foreach {
+ case CoalescedPartitionSpec(startReducerIndex, endReducerIndex, _) =>
+ var newPartitionDataSize = 0L
+ for (i <- startReducerIndex until endReducerIndex) {
+ if (i < originalBytesByPartition.length) {
+ newPartitionDataSize += originalBytesByPartition(i)
+ } else {
+ newPartitionDataSize += 0L
+ }
+ }
+ aqeReadPartitionsStats += newPartitionDataSize
+ case PartialReducerPartitionSpec(reducerIndex, startMapIndex, endMapIndex, _) =>
+ // over estimated stats
+ if (reducerIndex < originalBytesByPartition.length) {
+ aqeReadPartitionsStats += originalBytesByPartition(reducerIndex)
+ } else {
+ aqeReadPartitionsStats += 0L
+ }
+ case PartialMapperPartitionSpec(mapIndex, startReducerIndex, endReducerIndex) =>
+ // unknown stats
+ aqeReadPartitionsStats += 0L
+ case CoalescedMapperPartitionSpec(startMapIndex, endMapIndex, numReducers) =>
+ // unknown stats
+ aqeReadPartitionsStats += 0L
+ }
+ InputStats(known = true, totalBytes, rowCount, aqeReadPartitionsStats.toArray, ShuffleStats)
+ }
+ def createInputStats(logicalLink: Option[LogicalPlan]): Option[InputStats] = {
+ if (logicalLink.isEmpty) {
+ logInfo(s"no logicalLink associated")
+ return None
+ }
+ val logicalPlan = logicalLink.get
+ logInfo("scan logical link:" + logicalPlan)
+ logInfo("FileSourceScanTransformer logical plan " + logicalPlan)
+ val maybeCatalogTable = logicalPlan.collectFirst {
+ case LogicalRelation(_, _, catalogTable, _) => catalogTable
+ }
+ if (
+ maybeCatalogTable.isDefined && maybeCatalogTable.get.isDefined &&
+ maybeCatalogTable.get.get.stats.isDefined
+ ) {
+ val scanStats = maybeCatalogTable.get.get.stats.get
+ logInfo("catalogTable " + scanStats)
+ Some(
+ InputStats(
+ known = true,
+ scanStats.sizeInBytes,
+ scanStats.rowCount.getOrElse(0),
+ Array(),
+ ScanInputStats))
+ } else {
+ None
+ }
+ }
+}
diff --git a/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ColumnarCollapseTransformStages.scala b/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ColumnarCollapseTransformStages.scala
index c81d380a78..f35aea8fdc 100644
--- a/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ColumnarCollapseTransformStages.scala
+++ b/gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ColumnarCollapseTransformStages.scala
@@ -19,10 +19,11 @@ package org.apache.spark.sql.execution
import org.apache.gluten.backendsapi.BackendsApiManager
import org.apache.gluten.config.GlutenConfig
import org.apache.gluten.execution._
+import org.apache.gluten.extension.ApplyStageInputStatsRule
import org.apache.gluten.extension.columnar.transition.{Convention, ConventionReq}
import org.apache.gluten.metrics.MetricsUpdater
import org.apache.gluten.substrait.SubstraitContext
-import org.apache.gluten.substrait.rel.RelBuilder
+import org.apache.gluten.substrait.rel.{InputIteratorRelNode, RelBuilder}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.rdd.RDD
@@ -31,7 +32,7 @@ import org.apache.spark.sql.catalyst.expressions.{Attribute, SortOrder}
import org.apache.spark.sql.catalyst.plans.physical.Partitioning
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.util.truncatedString
-import org.apache.spark.sql.execution.adaptive.{BroadcastQueryStageExec, ShuffleQueryStageExec}
+import org.apache.spark.sql.execution.adaptive.{AQEShuffleReadExec, BroadcastQueryStageExec, BroadcastStats, InputStats, ShuffleQueryStageExec, ShuffleStageWrapper, ShuffleStats}
import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike, ShuffleExchangeLike}
import org.apache.spark.sql.execution.metric.SQLMetric
import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -76,6 +77,36 @@ case class InputIteratorTransformer(child: SparkPlan) extends UnaryTransformSupp
assert(child.isInstanceOf[ColumnarInputAdapter])
val operatorId = context.nextOperatorId(nodeName)
val readRel = RelBuilder.makeReadRelForInputIterator(child.output.asJava, context, operatorId)
+ if (GlutenConfig.get.enablePassStageInputStats) {
+ val inputStatsOpt: Option[InputStats] = {
+ val inputAdapter = child.asInstanceOf[ColumnarInputAdapter]
+ inputAdapter.child match {
+ case shuffle @ ShuffleStageWrapper(stats) =>
+ val inputStats = InputStats(
+ known = true,
+ sizeInBytes = stats.sizeInBytes,
+ rowCount = stats.rowCount.getOrElse(0),
+ bytesByPartitionId =
+ stats.mapOutputStatistics.map(_.bytesByPartitionId).getOrElse(Array()),
+ if (shuffle.isInstanceOf[BroadcastQueryStageExec]) BroadcastStats else ShuffleStats
+ )
+ shuffle match {
+ case a: AQEShuffleReadExec =>
+ Some(
+ ApplyStageInputStatsRule
+ .recomputeInputStatsForAQEShuffleReadExec(inputStats, a.partitionSpecs))
+ case _ => Some(inputStats)
+ }
+ case other =>
+ logInfo(s"hit InputIteratorTransformer ${other.getClass.toString} with stats")
+ None
+ }
+ }
+ if (inputStatsOpt.isDefined) {
+ logInfo(s"set input iterator stats ${inputStatsOpt.get}")
+ readRel.asInstanceOf[InputIteratorRelNode].setInputStats(inputStatsOpt.get)
+ }
+ }
TransformContext(output, readRel)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 29/38: [GLUTEN][CORE] Share metrics updaters in backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit fda31ae5c4b309bae1bcb59f3d23d7d3cfd3f6e9
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 20:34:13 2026 +0800
[GLUTEN][CORE] Share metrics updaters in backends-core
Lift 11 MetricsUpdater Scala classes that are identical between Velox-
and Bolt-style backends (after normalising backend naming) into
backends-core:
Expand / Filter / Generate / Limit / NestedLoopJoin / Project /
Sample / Sort / Union / Window / WriteFiles
Their only dependencies are MetricsUpdater (gluten-substrait),
IOperatorMetrics (gluten-substrait), and the now-shared
OperatorMetrics POJO in backends-core. No behaviour change: the
package path stays the same so backend metric-API wiring keeps working
unchanged.
---
.../src/main/scala/org/apache/gluten/metrics/ExpandMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/GenerateMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/LimitMetricsUpdater.scala | 0
.../scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/ProjectMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/SampleMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/SortMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/UnionMetricsUpdater.scala | 0
.../src/main/scala/org/apache/gluten/metrics/WindowMetricsUpdater.scala | 0
.../main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala | 0
11 files changed, 0 insertions(+), 0 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/ExpandMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/ExpandMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/ExpandMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/ExpandMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/FilterMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/GenerateMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/GenerateMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/GenerateMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/GenerateMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/LimitMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/LimitMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/LimitMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/LimitMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/ProjectMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/ProjectMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/ProjectMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/ProjectMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/SampleMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/SampleMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/SampleMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/SampleMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/SortMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/SortMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/SortMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/SortMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/UnionMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/UnionMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/UnionMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/UnionMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/WindowMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/WindowMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/WindowMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/WindowMetricsUpdater.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
rename to backends-core/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 21/38: [GLUTEN][CORE] Extract shared Substrait extension registry
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 91d67dc30d65697ff0456b50a8043e06aea091d7
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:37:32 2026 +0800
[GLUTEN][CORE] Extract shared Substrait extension registry
Move the duplicated Substrait extension-function registry logic from the Velox
collector into cpp/core so the first-tier Bolt/Velox shared patch can land as a
single common implementation.
Also keep the collector's VeloxSubstraitSignature include after the extraction
so TypePtr remains visible in the header and the patch commit records the
follow-up compile fix together with the main refactor.
---
cpp/core/CMakeLists.txt | 1 +
.../substrait/SubstraitExtensionRegistry.cc} | 43 ++++++--------
.../substrait/SubstraitExtensionRegistry.h} | 24 +++-----
cpp/velox/substrait/SubstraitExtensionCollector.cc | 39 ++----------
cpp/velox/substrait/SubstraitExtensionCollector.h | 69 ++--------------------
5 files changed, 37 insertions(+), 139 deletions(-)
diff --git a/cpp/core/CMakeLists.txt b/cpp/core/CMakeLists.txt
index 9530d603f5..d976e92bd0 100644
--- a/cpp/core/CMakeLists.txt
+++ b/cpp/core/CMakeLists.txt
@@ -121,6 +121,7 @@ set(SPARK_COLUMNAR_PLUGIN_SRCS
compute/Runtime.cc
compute/ProtobufUtils.cc
compute/ResultIterator.cc
+ substrait/SubstraitExtensionRegistry.cc
config/GlutenConfig.cc
jni/JniWrapper.cc
memory/AllocationListener.cc
diff --git a/cpp/velox/substrait/SubstraitExtensionCollector.cc b/cpp/core/substrait/SubstraitExtensionRegistry.cc
similarity index 67%
copy from cpp/velox/substrait/SubstraitExtensionCollector.cc
copy to cpp/core/substrait/SubstraitExtensionRegistry.cc
index 472ef04e3f..1dd00933cf 100644
--- a/cpp/velox/substrait/SubstraitExtensionCollector.cc
+++ b/cpp/core/substrait/SubstraitExtensionRegistry.cc
@@ -15,21 +15,12 @@
* limitations under the License.
*/
-#include "SubstraitExtensionCollector.h"
+#include "SubstraitExtensionRegistry.h"
namespace gluten {
-int SubstraitExtensionCollector::getReferenceNumber(
- const std::string& functionName,
- const std::vector<TypePtr>& arguments) {
- const auto& substraitFunctionSignature = VeloxSubstraitSignature::toSubstraitSignature(functionName, arguments);
- // TODO: Currently we treat all velox registry based function signatures as
- // custom substrait extension, so no uri link and leave it as empty.
- return getReferenceNumber({"", substraitFunctionSignature});
-}
-
template <typename T>
-bool SubstraitExtensionCollector::BiDirectionHashMap<T>::putIfAbsent(const int& key, const T& value) {
+bool SubstraitExtensionRegistry::BiDirectionHashMap<T>::putIfAbsent(const int& key, const T& value) {
if (forwardMap_.find(key) == forwardMap_.end() && reverseMap_.find(value) == reverseMap_.end()) {
forwardMap_[key] = value;
reverseMap_[value] = key;
@@ -38,7 +29,21 @@ bool SubstraitExtensionCollector::BiDirectionHashMap<T>::putIfAbsent(const int&
return false;
}
-void SubstraitExtensionCollector::addExtensionsToPlan(::substrait::Plan* plan) const {
+SubstraitExtensionRegistry::SubstraitExtensionRegistry() {
+ extensionFunctions_ = std::make_shared<BiDirectionHashMap<ExtensionFunctionId>>();
+}
+
+int SubstraitExtensionRegistry::getReferenceNumber(const ExtensionFunctionId& extensionFunctionId) {
+ const auto& extensionFunctionAnchorIt = extensionFunctions_->reverseMap().find(extensionFunctionId);
+ if (extensionFunctionAnchorIt != extensionFunctions_->reverseMap().end()) {
+ return extensionFunctionAnchorIt->second;
+ }
+ ++functionReferenceNumber_;
+ extensionFunctions_->putIfAbsent(functionReferenceNumber_, extensionFunctionId);
+ return functionReferenceNumber_;
+}
+
+void SubstraitExtensionRegistry::addExtensionsToPlan(::substrait::Plan* plan) const {
using SimpleExtensionURI = ::substrait::extensions::SimpleExtensionURI;
// Currently we don't introduce any substrait extension YAML files, so always
// only have one URI.
@@ -53,18 +58,4 @@ void SubstraitExtensionCollector::addExtensionsToPlan(::substrait::Plan* plan) c
}
}
-SubstraitExtensionCollector::SubstraitExtensionCollector() {
- extensionFunctions_ = std::make_shared<BiDirectionHashMap<ExtensionFunctionId>>();
-}
-
-int SubstraitExtensionCollector::getReferenceNumber(const ExtensionFunctionId& extensionFunctionId) {
- const auto& extensionFunctionAnchorIt = extensionFunctions_->reverseMap().find(extensionFunctionId);
- if (extensionFunctionAnchorIt != extensionFunctions_->reverseMap().end()) {
- return extensionFunctionAnchorIt->second;
- }
- ++functionReferenceNumber;
- extensionFunctions_->putIfAbsent(functionReferenceNumber, extensionFunctionId);
- return functionReferenceNumber;
-}
-
} // namespace gluten
diff --git a/cpp/velox/substrait/SubstraitExtensionCollector.h b/cpp/core/substrait/SubstraitExtensionRegistry.h
similarity index 82%
copy from cpp/velox/substrait/SubstraitExtensionCollector.h
copy to cpp/core/substrait/SubstraitExtensionRegistry.h
index 32cb25c92a..3e023ee930 100644
--- a/cpp/velox/substrait/SubstraitExtensionCollector.h
+++ b/cpp/core/substrait/SubstraitExtensionRegistry.h
@@ -17,13 +17,12 @@
#pragma once
-#include <optional>
-#include "VeloxSubstraitSignature.h"
+#include <memory>
+#include <string>
+#include <unordered_map>
+
#include "substrait/algebra.pb.h"
#include "substrait/plan.pb.h"
-#include "velox/core/Expressions.h"
-#include "velox/core/PlanNode.h"
-#include "velox/type/Type.h"
namespace gluten {
@@ -45,13 +44,11 @@ struct ExtensionFunctionId {
};
/// Assigns unique IDs to function signatures using ExtensionFunctionId.
-class SubstraitExtensionCollector {
+class SubstraitExtensionRegistry {
public:
- SubstraitExtensionCollector();
+ SubstraitExtensionRegistry();
- /// Given a scalar function name and argument types, return the functionId
- /// using ExtensionFunctionId.
- int getReferenceNumber(const std::string& functionName, const std::vector<TypePtr>& arguments);
+ int getReferenceNumber(const ExtensionFunctionId& extensionFunctionId);
/// Add extension functions to Substrait plan.
void addExtensionsToPlan(::substrait::Plan* plan) const;
@@ -81,15 +78,10 @@ class SubstraitExtensionCollector {
std::unordered_map<T, int> reverseMap_;
};
- /// Assigns unique IDs to function signatures using ExtensionFunctionId.
- int getReferenceNumber(const ExtensionFunctionId& extensionFunctionId);
-
- int functionReferenceNumber = -1;
+ int functionReferenceNumber_ = -1;
std::shared_ptr<BiDirectionHashMap<ExtensionFunctionId>> extensionFunctions_;
};
-using SubstraitExtensionCollectorPtr = std::shared_ptr<SubstraitExtensionCollector>;
-
} // namespace gluten
namespace std {
diff --git a/cpp/velox/substrait/SubstraitExtensionCollector.cc b/cpp/velox/substrait/SubstraitExtensionCollector.cc
index 472ef04e3f..1a4a971a91 100644
--- a/cpp/velox/substrait/SubstraitExtensionCollector.cc
+++ b/cpp/velox/substrait/SubstraitExtensionCollector.cc
@@ -17,6 +17,8 @@
#include "SubstraitExtensionCollector.h"
+#include "VeloxSubstraitSignature.h"
+
namespace gluten {
int SubstraitExtensionCollector::getReferenceNumber(
@@ -25,46 +27,15 @@ int SubstraitExtensionCollector::getReferenceNumber(
const auto& substraitFunctionSignature = VeloxSubstraitSignature::toSubstraitSignature(functionName, arguments);
// TODO: Currently we treat all velox registry based function signatures as
// custom substrait extension, so no uri link and leave it as empty.
- return getReferenceNumber({"", substraitFunctionSignature});
-}
-
-template <typename T>
-bool SubstraitExtensionCollector::BiDirectionHashMap<T>::putIfAbsent(const int& key, const T& value) {
- if (forwardMap_.find(key) == forwardMap_.end() && reverseMap_.find(value) == reverseMap_.end()) {
- forwardMap_[key] = value;
- reverseMap_[value] = key;
- return true;
- }
- return false;
+ return extensionRegistry_->getReferenceNumber({"", substraitFunctionSignature});
}
void SubstraitExtensionCollector::addExtensionsToPlan(::substrait::Plan* plan) const {
- using SimpleExtensionURI = ::substrait::extensions::SimpleExtensionURI;
- // Currently we don't introduce any substrait extension YAML files, so always
- // only have one URI.
- SimpleExtensionURI* extensionUri = plan->add_extension_uris();
- extensionUri->set_extension_uri_anchor(1);
-
- for (const auto& [referenceNum, functionId] : extensionFunctions_->forwardMap()) {
- auto extensionFunction = plan->add_extensions()->mutable_extension_function();
- extensionFunction->set_extension_uri_reference(extensionUri->extension_uri_anchor());
- extensionFunction->set_function_anchor(referenceNum);
- extensionFunction->set_name(functionId.signature);
- }
+ extensionRegistry_->addExtensionsToPlan(plan);
}
SubstraitExtensionCollector::SubstraitExtensionCollector() {
- extensionFunctions_ = std::make_shared<BiDirectionHashMap<ExtensionFunctionId>>();
-}
-
-int SubstraitExtensionCollector::getReferenceNumber(const ExtensionFunctionId& extensionFunctionId) {
- const auto& extensionFunctionAnchorIt = extensionFunctions_->reverseMap().find(extensionFunctionId);
- if (extensionFunctionAnchorIt != extensionFunctions_->reverseMap().end()) {
- return extensionFunctionAnchorIt->second;
- }
- ++functionReferenceNumber;
- extensionFunctions_->putIfAbsent(functionReferenceNumber, extensionFunctionId);
- return functionReferenceNumber;
+ extensionRegistry_ = std::make_shared<SubstraitExtensionRegistry>();
}
} // namespace gluten
diff --git a/cpp/velox/substrait/SubstraitExtensionCollector.h b/cpp/velox/substrait/SubstraitExtensionCollector.h
index 32cb25c92a..4faa9e6f67 100644
--- a/cpp/velox/substrait/SubstraitExtensionCollector.h
+++ b/cpp/velox/substrait/SubstraitExtensionCollector.h
@@ -17,33 +17,18 @@
#pragma once
-#include <optional>
+#include <memory>
+#include <string>
+#include <vector>
+
#include "VeloxSubstraitSignature.h"
-#include "substrait/algebra.pb.h"
-#include "substrait/plan.pb.h"
+#include "substrait/SubstraitExtensionRegistry.h"
#include "velox/core/Expressions.h"
#include "velox/core/PlanNode.h"
#include "velox/type/Type.h"
namespace gluten {
-struct ExtensionFunctionId {
- /// Substrait extension YAML file uri.
- std::string uri;
-
- /// Substrait signature used in the function extension declaration is a
- /// combination of the name of the function along with a list of input
- /// argument types.The format is as follows : <function
- /// name>:<short_arg_type0>_<short_arg_type1>_..._<short_arg_typeN> for more
- /// detail information about the argument type please refer to link
- /// https://substrait.io/extensions/#function-signature-compound-names.
- std::string signature;
-
- bool operator==(const ExtensionFunctionId& other) const {
- return (uri == other.uri && signature == other.signature);
- }
-};
-
/// Assigns unique IDs to function signatures using ExtensionFunctionId.
class SubstraitExtensionCollector {
public:
@@ -57,51 +42,9 @@ class SubstraitExtensionCollector {
void addExtensionsToPlan(::substrait::Plan* plan) const;
private:
- /// A bi-direction hash map to keep the relation between reference number and
- /// either function or type signature.
- template <class T>
- class BiDirectionHashMap {
- public:
- /// Add (key, value) pair if doesn't exist already, i.e. forwardMap doesn't
- /// contain the key and reverseMap doesn't contain the value.
- ///
- /// @return True if the values were added successfully. False, otherwise.
- bool putIfAbsent(const int& key, const T& value);
-
- const std::unordered_map<int, ExtensionFunctionId> forwardMap() const {
- return forwardMap_;
- }
-
- const std::unordered_map<T, int>& reverseMap() const {
- return reverseMap_;
- }
-
- private:
- std::unordered_map<int, T> forwardMap_;
- std::unordered_map<T, int> reverseMap_;
- };
-
- /// Assigns unique IDs to function signatures using ExtensionFunctionId.
- int getReferenceNumber(const ExtensionFunctionId& extensionFunctionId);
-
- int functionReferenceNumber = -1;
- std::shared_ptr<BiDirectionHashMap<ExtensionFunctionId>> extensionFunctions_;
+ std::shared_ptr<SubstraitExtensionRegistry> extensionRegistry_;
};
using SubstraitExtensionCollectorPtr = std::shared_ptr<SubstraitExtensionCollector>;
} // namespace gluten
-
-namespace std {
-
-/// Hash function of gluten::ExtensionFunctionId.
-template <>
-struct hash<gluten::ExtensionFunctionId> {
- size_t operator()(const gluten::ExtensionFunctionId& k) const {
- size_t val = hash<std::string>()(k.uri);
- val = val * 31 + hash<std::string>()(k.signature);
- return val;
- }
-};
-
-}; // namespace std
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 24/38: [GLUTEN][CORE] Share JVM test input resources
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 9b1f9f828fdb2179aed63ee0129138d5b26b31ba
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 15:26:07 2026 +0800
[GLUTEN][CORE] Share JVM test input resources
Move duplicated Velox test input data into gluten-substrait test resources and
switch the remaining Velox suites to load TPCH parquet data from the shared
classpath location.
---
.../org/apache/gluten/execution/VeloxTPCHDeltaSuite.scala | 2 +-
.../org/apache/gluten/execution/VeloxTPCHHudiSuite.scala | 2 +-
.../gluten/execution/TestTPCHStoragePartitionedJoins.java | 4 +---
.../org/apache/gluten/execution/VeloxTPCHIcebergSuite.scala | 2 +-
...f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet | Bin
...000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc | Bin
...f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet | Bin
...000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc | Bin
.../src/test/resources/datasource/csv/student.csv | 0
.../src/test/resources/datasource/csv/student_option.csv | 0
.../test/resources/datasource/csv/student_option_schema.csv | 0
.../test/resources/datasource/csv/student_option_str.csv | 0
.../test/resources/parquet-for-read/test-append_1.parquet | Bin
.../test/resources/parquet-for-read/test-append_2.parquet | Bin
.../parquet-for-read/test-empty-row-group_1.parquet | Bin
.../parquet-for-read/test-empty-row-group_2.parquet | Bin
.../parquet-for-read/test-empty-row-group_3.parquet | Bin
.../test-file-with-no-column-indexes-1.parquet | Bin
...a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet | Bin
...6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet | Bin
...8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet | Bin
...41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet | Bin
...e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet | Bin
...2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet | Bin
...90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet | Bin
...0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet | Bin
26 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/backends-velox/src-delta/test/scala/org/apache/gluten/execution/VeloxTPCHDeltaSuite.scala b/backends-velox/src-delta/test/scala/org/apache/gluten/execution/VeloxTPCHDeltaSuite.scala
index 238f5ca56f..036080a347 100644
--- a/backends-velox/src-delta/test/scala/org/apache/gluten/execution/VeloxTPCHDeltaSuite.scala
+++ b/backends-velox/src-delta/test/scala/org/apache/gluten/execution/VeloxTPCHDeltaSuite.scala
@@ -25,7 +25,7 @@ class VeloxTPCHDeltaSuite extends VeloxTPCHSuite {
getClass.getResource("/").getPath + "../../../src/test/resources"
override protected val resourcePath: String =
- new File(tpchBasePath, "tpch-data-parquet").getCanonicalPath
+ getClass.getResource("/tpch-data-parquet").getFile
override protected val queriesResults: String =
new File(tpchBasePath, "queries-output").getCanonicalPath
diff --git a/backends-velox/src-hudi/test/scala/org/apache/gluten/execution/VeloxTPCHHudiSuite.scala b/backends-velox/src-hudi/test/scala/org/apache/gluten/execution/VeloxTPCHHudiSuite.scala
index b5680a9c2c..734ff5606e 100644
--- a/backends-velox/src-hudi/test/scala/org/apache/gluten/execution/VeloxTPCHHudiSuite.scala
+++ b/backends-velox/src-hudi/test/scala/org/apache/gluten/execution/VeloxTPCHHudiSuite.scala
@@ -25,7 +25,7 @@ class VeloxTPCHHudiSuite extends VeloxTPCHSuite {
getClass.getResource("/").getPath + "../../../src/test/resources"
override protected val resourcePath: String =
- new File(tpchBasePath, "tpch-data-parquet").getCanonicalPath
+ getClass.getResource("/tpch-data-parquet").getFile
override protected val queriesResults: String =
new File(tpchBasePath, "queries-output").getCanonicalPath
diff --git a/backends-velox/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java b/backends-velox/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java
index d2edc28fc8..cdab374053 100644
--- a/backends-velox/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java
+++ b/backends-velox/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java
@@ -79,11 +79,9 @@ public class TestTPCHStoragePartitionedJoins extends SparkTestBaseWithCatalog {
// we don't need to test the catalog, so only test the testhadoop catalog
@Before
public void createTPCHNotNullTables() {
+ final String tableDir = this.getClass().getResource("/tpch-data-parquet").getFile();
tableNames.forEach(
table -> {
- String tableDir = tpchBasePath + "/tpch-data-parquet";
- // String tableDir =
- // "/Users/chengchengjin/code/gluten/backends-velox/src/test/resources/tpch-data-parquet";
String tablePath = new File(tableDir, table).getAbsolutePath();
Dataset<Row> tableDF = spark.read().format("parquet").load(tablePath);
tableDF.createOrReplaceTempView(PARQUET_TABLE_PREFIX + table);
diff --git a/backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/VeloxTPCHIcebergSuite.scala b/backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/VeloxTPCHIcebergSuite.scala
index 5456e0ba89..66d16a50a2 100644
--- a/backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/VeloxTPCHIcebergSuite.scala
+++ b/backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/VeloxTPCHIcebergSuite.scala
@@ -28,7 +28,7 @@ class VeloxTPCHIcebergSuite extends VeloxTPCHSuite {
getClass.getResource("/").getPath + "../../../src/test/resources"
override protected val resourcePath: String =
- new File(tpchBasePath, "tpch-data-parquet").getCanonicalPath
+ getClass.getResource("/tpch-data-parquet").getFile
// FIXME: Unused.
override protected val queriesResults: String =
diff --git a/backends-velox/src/test/resources/data-type-validation-data/type1/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet b/gluten-substrait/src/test/resources/data-type-validation-data/type1/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/data-type-validation-data/type1/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/data-type-validation-data/type1/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/data-type-validation-data/type1_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc b/gluten-substrait/src/test/resources/data-type-validation-data/type1_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc
similarity index 100%
rename from backends-velox/src/test/resources/data-type-validation-data/type1_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc
rename to gluten-substrait/src/test/resources/data-type-validation-data/type1_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc
diff --git a/backends-velox/src/test/resources/data-type-validation-data/type2/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet b/gluten-substrait/src/test/resources/data-type-validation-data/type2/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/data-type-validation-data/type2/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/data-type-validation-data/type2/part-00000-f401debc-29d1-47b6-82c2-051bd0e12df1-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/data-type-validation-data/type2_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc b/gluten-substrait/src/test/resources/data-type-validation-data/type2_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc
similarity index 100%
rename from backends-velox/src/test/resources/data-type-validation-data/type2_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc
rename to gluten-substrait/src/test/resources/data-type-validation-data/type2_orc/part-00000-e084aa04-c639-40f9-a85f-4b8b10dae1b4-c000.snappy.orc
diff --git a/backends-velox/src/test/resources/datasource/csv/student.csv b/gluten-substrait/src/test/resources/datasource/csv/student.csv
similarity index 100%
rename from backends-velox/src/test/resources/datasource/csv/student.csv
rename to gluten-substrait/src/test/resources/datasource/csv/student.csv
diff --git a/backends-velox/src/test/resources/datasource/csv/student_option.csv b/gluten-substrait/src/test/resources/datasource/csv/student_option.csv
similarity index 100%
rename from backends-velox/src/test/resources/datasource/csv/student_option.csv
rename to gluten-substrait/src/test/resources/datasource/csv/student_option.csv
diff --git a/backends-velox/src/test/resources/datasource/csv/student_option_schema.csv b/gluten-substrait/src/test/resources/datasource/csv/student_option_schema.csv
similarity index 100%
rename from backends-velox/src/test/resources/datasource/csv/student_option_schema.csv
rename to gluten-substrait/src/test/resources/datasource/csv/student_option_schema.csv
diff --git a/backends-velox/src/test/resources/datasource/csv/student_option_str.csv b/gluten-substrait/src/test/resources/datasource/csv/student_option_str.csv
similarity index 100%
rename from backends-velox/src/test/resources/datasource/csv/student_option_str.csv
rename to gluten-substrait/src/test/resources/datasource/csv/student_option_str.csv
diff --git a/backends-velox/src/test/resources/parquet-for-read/test-append_1.parquet b/gluten-substrait/src/test/resources/parquet-for-read/test-append_1.parquet
similarity index 100%
rename from backends-velox/src/test/resources/parquet-for-read/test-append_1.parquet
rename to gluten-substrait/src/test/resources/parquet-for-read/test-append_1.parquet
diff --git a/backends-velox/src/test/resources/parquet-for-read/test-append_2.parquet b/gluten-substrait/src/test/resources/parquet-for-read/test-append_2.parquet
similarity index 100%
rename from backends-velox/src/test/resources/parquet-for-read/test-append_2.parquet
rename to gluten-substrait/src/test/resources/parquet-for-read/test-append_2.parquet
diff --git a/backends-velox/src/test/resources/parquet-for-read/test-empty-row-group_1.parquet b/gluten-substrait/src/test/resources/parquet-for-read/test-empty-row-group_1.parquet
similarity index 100%
rename from backends-velox/src/test/resources/parquet-for-read/test-empty-row-group_1.parquet
rename to gluten-substrait/src/test/resources/parquet-for-read/test-empty-row-group_1.parquet
diff --git a/backends-velox/src/test/resources/parquet-for-read/test-empty-row-group_2.parquet b/gluten-substrait/src/test/resources/parquet-for-read/test-empty-row-group_2.parquet
similarity index 100%
rename from backends-velox/src/test/resources/parquet-for-read/test-empty-row-group_2.parquet
rename to gluten-substrait/src/test/resources/parquet-for-read/test-empty-row-group_2.parquet
diff --git a/backends-velox/src/test/resources/parquet-for-read/test-empty-row-group_3.parquet b/gluten-substrait/src/test/resources/parquet-for-read/test-empty-row-group_3.parquet
similarity index 100%
rename from backends-velox/src/test/resources/parquet-for-read/test-empty-row-group_3.parquet
rename to gluten-substrait/src/test/resources/parquet-for-read/test-empty-row-group_3.parquet
diff --git a/backends-velox/src/test/resources/parquet-for-read/test-file-with-no-column-indexes-1.parquet b/gluten-substrait/src/test/resources/parquet-for-read/test-file-with-no-column-indexes-1.parquet
similarity index 100%
rename from backends-velox/src/test/resources/parquet-for-read/test-file-with-no-column-indexes-1.parquet
rename to gluten-substrait/src/test/resources/parquet-for-read/test-file-with-no-column-indexes-1.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/customer/part-00000-a1a42661-7a85-42da-b831-f489a5545d61-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/nation/part-00000-8a852c7b-da96-4425-9ce8-d188f97f3afe-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/orders/part-00000-41cc94ee-a98d-4103-83b1-0b5bcebe03eb-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/part/part-00000-e664e655-0ada-4e0b-a8aa-213d1e4c3797-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/partsupp/part-00000-2c7e524c-3f91-4228-b003-64ad2c8059a1-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/region/part-00000-90d7c5d9-46eb-4bf3-8f00-6eee7436761d-c000.snappy.parquet
diff --git a/backends-velox/src/test/resources/tpch-data-parquet/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet b/gluten-substrait/src/test/resources/tpch-data-parquet/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet
similarity index 100%
rename from backends-velox/src/test/resources/tpch-data-parquet/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet
rename to gluten-substrait/src/test/resources/tpch-data-parquet/supplier/part-00000-0a763951-8a7b-4d7f-a34a-c97e9b4859f0-c000.snappy.parquet
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 02/38: [GLUTEN][VL] Support sequence function in Velox backend
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit cdbbb62a8894ed2440eaec5cb3f5fe060c6c5093
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Fri Jun 26 15:59:01 2026 +0800
[GLUTEN][VL] Support sequence function in Velox backend
Remove "sequence" from the validator black list so that the Spark
sequence expression can be offloaded to Velox, which natively supports
sequence for integral/date/timestamp inputs. Add a UT covering column
inputs (non constant-folding) and update the scalar function support doc.
Generated-by: TraeCli openrouter-3o
---
.../gluten/functions/ScalarFunctionsValidateSuite.scala | 11 +++++++++++
cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc | 2 +-
docs/velox-backend-scalar-function-support.md | 4 ++--
3 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala b/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
index 74b2933a67..38db4e9b3a 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
+++ b/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
@@ -540,6 +540,17 @@ abstract class ScalarFunctionsValidateSuite extends FunctionsValidateSuite {
}
}
+ test("sequence function with column inputs offloaded to native") {
+ runQueryAndCompare("""SELECT sequence(1, l_linenumber), l_orderkey
+ | from lineitem limit 100""".stripMargin) {
+ checkGlutenPlan[ProjectExecTransformer]
+ }
+ runQueryAndCompare("""SELECT sequence(l_linenumber, l_linenumber + 5, 2), l_orderkey
+ | from lineitem limit 100""".stripMargin) {
+ checkGlutenPlan[ProjectExecTransformer]
+ }
+ }
+
test("map_from_arrays optimized by Spark constant folding") {
withSQLConf(("spark.sql.optimizer.excludedRules", "")) {
runQueryAndCompare("""SELECT map_from_arrays(sequence(1, 5),sequence(1, 5)), l_orderkey
diff --git a/cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc b/cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc
index 767af98c76..75d691bea5 100644
--- a/cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc
+++ b/cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc
@@ -56,7 +56,7 @@ const char* extractFileName(const char* file) {
const std::unordered_set<std::string> kRegexFunctions =
{"regexp_extract", "regexp_extract_all", "regexp_replace", "rlike", "split"};
-const std::unordered_set<std::string> kBlackList = {"split_part", "sequence", "approx_percentile", "map_from_arrays"};
+const std::unordered_set<std::string> kBlackList = {"split_part", "approx_percentile", "map_from_arrays"};
} // namespace
bool SubstraitToVeloxPlanValidator::parseVeloxType(
diff --git a/docs/velox-backend-scalar-function-support.md b/docs/velox-backend-scalar-function-support.md
index e16baf3701..1da7586a6d 100644
--- a/docs/velox-backend-scalar-function-support.md
+++ b/docs/velox-backend-scalar-function-support.md
@@ -1,6 +1,6 @@
# Scalar Functions Support Status
-**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 242 functions and partially supports 25 functions.**
+**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 243 functions and partially supports 25 functions.**
## Array Functions
@@ -26,7 +26,7 @@
| arrays_zip | ArraysZip | S | |
| flatten | Flatten | S | |
| get | Get | S | |
-| sequence | Sequence | | |
+| sequence | Sequence | S | |
| shuffle | Shuffle | S | |
| slice | Slice | S | |
| sort_array | SortArray | S | |
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 26/38: [GLUTEN][CORE] Share WholeStageTransformerSuite under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 3d4e02631a05d694e11b5904585e1197d8dec0da
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 20:23:59 2026 +0800
[GLUTEN][CORE] Share WholeStageTransformerSuite under backends-core
Extract the `disableFallbackCheck` thin shim used by all backend test
suites into a common SharedWholeStageTransformerSuite under
backends-core/src/test, so future backends only need to subclass it
without duplicating the same one-liner.
Velox keeps `VeloxWholeStageTransformerSuite` as a thin alias to keep
call sites unchanged.
Generated-by: TraeCli openrouter-3o
---
.../apache/gluten/execution/SharedWholeStageTransformerSuite.scala | 2 +-
.../org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala | 4 +---
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala b/backends-core/src/test/scala/org/apache/gluten/execution/SharedWholeStageTransformerSuite.scala
similarity index 91%
copy from backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala
copy to backends-core/src/test/scala/org/apache/gluten/execution/SharedWholeStageTransformerSuite.scala
index f393d4a751..bcec402359 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala
+++ b/backends-core/src/test/scala/org/apache/gluten/execution/SharedWholeStageTransformerSuite.scala
@@ -16,6 +16,6 @@
*/
package org.apache.gluten.execution
-abstract class VeloxWholeStageTransformerSuite extends WholeStageTransformerSuite {
+abstract class SharedWholeStageTransformerSuite extends WholeStageTransformerSuite {
disableFallbackCheck
}
diff --git a/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala b/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala
index f393d4a751..2422530ba2 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala
+++ b/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxWholeStageTransformerSuite.scala
@@ -16,6 +16,4 @@
*/
package org.apache.gluten.execution
-abstract class VeloxWholeStageTransformerSuite extends WholeStageTransformerSuite {
- disableFallbackCheck
-}
+abstract class VeloxWholeStageTransformerSuite extends SharedWholeStageTransformerSuite
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 23/38: [GLUTEN][CORE] Extract shared VariantToVectorConverter and Jol filesystem registration into cpp/core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit abe3f09666d6b1e98e8febf18d29f925154e3126
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 11:19:43 2026 +0800
[GLUTEN][CORE] Extract shared VariantToVectorConverter and Jol filesystem registration into cpp/core
Move the backend-agnostic parts of VariantToVectorConverter and the
registerJolFileSystem declaration into cpp/core so future backend rebases
don't conflict on these files. The Velox backend keeps its original entry
names and only thin-wraps the shared implementation (backend-specific
dispatch / unsupported macros are injected via GLUTEN_* aliases), so Velox
behavior is unchanged.
Generated-by: TraeCli openrouter-3o
---
cpp/{velox => core}/jni/JniFileSystem.h | 9 +---
.../substrait/VariantToVectorConverter.cc | 13 +++--
.../substrait/VariantToVectorConverter.h | 6 +--
cpp/velox/jni/JniFileSystem.h | 7 +--
cpp/velox/substrait/VariantToVectorConverter.cc | 55 +++-------------------
cpp/velox/substrait/VariantToVectorConverter.h | 8 +---
6 files changed, 22 insertions(+), 76 deletions(-)
diff --git a/cpp/velox/jni/JniFileSystem.h b/cpp/core/jni/JniFileSystem.h
similarity index 86%
copy from cpp/velox/jni/JniFileSystem.h
copy to cpp/core/jni/JniFileSystem.h
index 11c0f27b36..ada83b68a7 100644
--- a/cpp/velox/jni/JniFileSystem.h
+++ b/cpp/core/jni/JniFileSystem.h
@@ -17,10 +17,7 @@
#pragma once
-#include <jni.h>
-
-#include "velox/common/file/File.h"
-#include "velox/common/file/FileSystems.h"
+#include <cstdint>
namespace gluten {
@@ -29,8 +26,4 @@ namespace gluten {
// local fs directly.
void registerJolFileSystem(uint64_t maxFileSize);
-void initVeloxJniFileSystem(JNIEnv* env);
-
-void finalizeVeloxJniFileSystem(JNIEnv* env);
-
} // namespace gluten
diff --git a/cpp/velox/substrait/VariantToVectorConverter.cc b/cpp/core/substrait/VariantToVectorConverter.cc
similarity index 82%
copy from cpp/velox/substrait/VariantToVectorConverter.cc
copy to cpp/core/substrait/VariantToVectorConverter.cc
index f33633b91e..86a80e8772 100644
--- a/cpp/velox/substrait/VariantToVectorConverter.cc
+++ b/cpp/core/substrait/VariantToVectorConverter.cc
@@ -16,7 +16,14 @@
*/
#include "VariantToVectorConverter.h"
-#include "velox/vector/FlatVector.h"
+
+#ifndef GLUTEN_DYNAMIC_SCALAR_TYPE_DISPATCH
+#error "GLUTEN_DYNAMIC_SCALAR_TYPE_DISPATCH must be defined before including VariantToVectorConverter.cc"
+#endif
+
+#ifndef GLUTEN_UNSUPPORTED
+#error "GLUTEN_UNSUPPORTED must be defined before including VariantToVectorConverter.cc"
+#endif
namespace gluten {
@@ -43,7 +50,7 @@ VectorPtr setVectorFromVariantsByKind<TypeKind::VARBINARY>(
const std::vector<variant>& /* values */,
const TypePtr& /*type*/,
memory::MemoryPool* /* pool */) {
- VELOX_UNSUPPORTED("Return of VARBINARY data is not supported");
+ GLUTEN_UNSUPPORTED("Return of VARBINARY data is not supported");
}
template <>
@@ -65,7 +72,7 @@ VectorPtr setVectorFromVariantsByKind<TypeKind::VARCHAR>(
} // namespace
VectorPtr setVectorFromVariants(const TypePtr& type, const std::vector<variant>& values, memory::MemoryPool* pool) {
- return VELOX_DYNAMIC_SCALAR_TYPE_DISPATCH(setVectorFromVariantsByKind, type->kind(), values, type, pool);
+ return GLUTEN_DYNAMIC_SCALAR_TYPE_DISPATCH(setVectorFromVariantsByKind, type->kind(), values, type, pool);
}
} // namespace gluten
diff --git a/cpp/velox/substrait/VariantToVectorConverter.h b/cpp/core/substrait/VariantToVectorConverter.h
similarity index 89%
copy from cpp/velox/substrait/VariantToVectorConverter.h
copy to cpp/core/substrait/VariantToVectorConverter.h
index 5d034884c9..5c27e0ad52 100644
--- a/cpp/velox/substrait/VariantToVectorConverter.h
+++ b/cpp/core/substrait/VariantToVectorConverter.h
@@ -17,13 +17,11 @@
#pragma once
-#include "velox/vector/BaseVector.h"
-
-using namespace facebook::velox;
+#include <vector>
namespace gluten {
-/// Create Base Vector from velox variants.
+/// Create Base Vector from backend variants.
/// Only scalar types are supported except VARBINARY.
VectorPtr setVectorFromVariants(const TypePtr& type, const std::vector<variant>& values, memory::MemoryPool* pool);
diff --git a/cpp/velox/jni/JniFileSystem.h b/cpp/velox/jni/JniFileSystem.h
index 11c0f27b36..c3a5f3351a 100644
--- a/cpp/velox/jni/JniFileSystem.h
+++ b/cpp/velox/jni/JniFileSystem.h
@@ -22,12 +22,9 @@
#include "velox/common/file/File.h"
#include "velox/common/file/FileSystems.h"
-namespace gluten {
+#include "../../core/jni/JniFileSystem.h"
-// Register JNI-or-local (or JVM-over-local, as long as it describes what happens here) file system. maxFileSize is
-// necessary (!= 0) because we use this size to decide whether a new file can fit in JVM heap, otherwise we write it via
-// local fs directly.
-void registerJolFileSystem(uint64_t maxFileSize);
+namespace gluten {
void initVeloxJniFileSystem(JNIEnv* env);
diff --git a/cpp/velox/substrait/VariantToVectorConverter.cc b/cpp/velox/substrait/VariantToVectorConverter.cc
index f33633b91e..aa80237a44 100644
--- a/cpp/velox/substrait/VariantToVectorConverter.cc
+++ b/cpp/velox/substrait/VariantToVectorConverter.cc
@@ -16,56 +16,13 @@
*/
#include "VariantToVectorConverter.h"
-#include "velox/vector/FlatVector.h"
-
-namespace gluten {
-
-namespace {
-template <TypeKind KIND>
-VectorPtr
-setVectorFromVariantsByKind(const std::vector<variant>& values, const TypePtr& type, memory::MemoryPool* pool) {
- using T = typename TypeTraits<KIND>::NativeType;
-
- auto flatVector = BaseVector::create<FlatVector<T>>(type, values.size(), pool);
- for (vector_size_t i = 0; i < values.size(); i++) {
- if (values[i].isNull()) {
- flatVector->setNull(i, true);
- } else {
- flatVector->set(i, values[i].value<T>());
- }
- }
- return flatVector;
-}
-
-template <>
-VectorPtr setVectorFromVariantsByKind<TypeKind::VARBINARY>(
- const std::vector<variant>& /* values */,
- const TypePtr& /*type*/,
- memory::MemoryPool* /* pool */) {
- VELOX_UNSUPPORTED("Return of VARBINARY data is not supported");
-}
-
-template <>
-VectorPtr setVectorFromVariantsByKind<TypeKind::VARCHAR>(
- const std::vector<variant>& values,
- const TypePtr& type,
- memory::MemoryPool* pool) {
- auto flatVector = BaseVector::create<FlatVector<StringView>>(type, values.size(), pool);
+#include "velox/vector/FlatVector.h"
- for (vector_size_t i = 0; i < values.size(); i++) {
- if (values[i].isNull()) {
- flatVector->setNull(i, true);
- } else {
- flatVector->set(i, StringView(values[i].value<TypeKind::VARCHAR>()));
- }
- }
- return flatVector;
-}
-} // namespace
+#define GLUTEN_DYNAMIC_SCALAR_TYPE_DISPATCH VELOX_DYNAMIC_SCALAR_TYPE_DISPATCH
+#define GLUTEN_UNSUPPORTED VELOX_UNSUPPORTED
-VectorPtr setVectorFromVariants(const TypePtr& type, const std::vector<variant>& values, memory::MemoryPool* pool) {
- return VELOX_DYNAMIC_SCALAR_TYPE_DISPATCH(setVectorFromVariantsByKind, type->kind(), values, type, pool);
-}
+#include "../../core/substrait/VariantToVectorConverter.cc"
-} // namespace gluten
+#undef GLUTEN_DYNAMIC_SCALAR_TYPE_DISPATCH
+#undef GLUTEN_UNSUPPORTED
diff --git a/cpp/velox/substrait/VariantToVectorConverter.h b/cpp/velox/substrait/VariantToVectorConverter.h
index 5d034884c9..262dab080e 100644
--- a/cpp/velox/substrait/VariantToVectorConverter.h
+++ b/cpp/velox/substrait/VariantToVectorConverter.h
@@ -21,10 +21,4 @@
using namespace facebook::velox;
-namespace gluten {
-
-/// Create Base Vector from velox variants.
-/// Only scalar types are supported except VARBINARY.
-VectorPtr setVectorFromVariants(const TypePtr& type, const std::vector<variant>& values, memory::MemoryPool* pool);
-
-} // namespace gluten
+#include "../../core/substrait/VariantToVectorConverter.h"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 10/38: [GLUTEN][CORE] Add backend hook for sort aggregate offload
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 90e224a833580f4673559be742e0412e83de1df0
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 21:18:02 2026 +0800
[GLUTEN][CORE] Add backend hook for sort aggregate offload
---
.../org/apache/gluten/backendsapi/SparkPlanExecApi.scala | 11 +++++++++++
.../extension/columnar/offload/OffloadSingleNodeRules.scala | 2 +-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
index 79f3d67c0e..ee347c183e 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala
@@ -37,6 +37,7 @@ import org.apache.spark.sql.catalyst.optimizer.BuildSide
import org.apache.spark.sql.catalyst.plans.JoinType
import org.apache.spark.sql.catalyst.plans.physical.{BroadcastMode, Partitioning}
import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
import org.apache.spark.sql.execution.datasources.FileFormat
import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
import org.apache.spark.sql.execution.joins.BuildSideRelation
@@ -102,6 +103,16 @@ trait SparkPlanExecApi {
resultExpressions,
child)
+ /**
+ * Offload a [[org.apache.spark.sql.execution.aggregate.SortAggregateExec]] to a columnar
+ * transformer. By default it goes through [[HashAggregateExecBaseTransformer.fromSortAggregate]]
+ * so that the result is tagged as a sort-based aggregate (used by sort elimination rules).
+ * Backends that do not need the sort-based semantic can override this to fall back to a regular
+ * hash aggregate transformer.
+ */
+ def offloadSortAggregate(plan: BaseAggregateExec): HashAggregateExecBaseTransformer =
+ HashAggregateExecBaseTransformer.fromSortAggregate(plan)
+
/** Generate HashAggregateExecPullOutHelper */
def genHashAggregateExecPullOutHelper(
aggregateExpressions: Seq[AggregateExpression],
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/offload/OffloadSingleNodeRules.scala b/gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/offload/OffloadSingleNodeRules.scala
index 3d844607b3..19c2fc40e7 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/offload/OffloadSingleNodeRules.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/offload/OffloadSingleNodeRules.scala
@@ -215,7 +215,7 @@ object OffloadOthers {
case plan: HashAggregateExec =>
HashAggregateExecBaseTransformer.from(plan)
case plan: SortAggregateExec =>
- HashAggregateExecBaseTransformer.fromSortAggregate(plan)
+ BackendsApiManager.getSparkPlanExecApiInstance.offloadSortAggregate(plan)
case plan: ObjectHashAggregateExec =>
HashAggregateExecBaseTransformer.from(plan)
case plan: UnionExec =>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 14/38: [GLUTEN][CORE] Extract shared write-file metrics updater
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit bbaf1c218edbcb8e5165884f883606c0ba66d5e7
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:01:03 2026 +0800
[GLUTEN][CORE] Extract shared write-file metrics updater
Introduce the backends-core module with a shared write-file metrics base so Velox and ClickHouse can reuse the same common SQLMetric updates while keeping backend-specific metrics in thin wrappers.
---
backends-clickhouse/pom.xml | 6 ++
.../gluten/metrics/WriteFilesMetricsUpdater.scala | 8 +-
backends-core/pom.xml | 95 ++++++++++++++++++++++
.../shared/SharedWriteFilesMetricsUpdater.scala | 18 ++--
backends-velox/pom.xml | 6 ++
.../gluten/metrics/WriteFilesMetricsUpdater.scala | 8 +-
pom.xml | 1 +
7 files changed, 128 insertions(+), 14 deletions(-)
diff --git a/backends-clickhouse/pom.xml b/backends-clickhouse/pom.xml
index 816aeab6f8..1743036681 100644
--- a/backends-clickhouse/pom.xml
+++ b/backends-clickhouse/pom.xml
@@ -46,6 +46,12 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>backends-core</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
diff --git a/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala b/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
index 5a04b40433..91cfbbccfd 100644
--- a/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
+++ b/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
@@ -16,15 +16,17 @@
*/
package org.apache.gluten.metrics
+import org.apache.gluten.metrics.shared.SharedWriteFilesMetricsUpdater
+
import org.apache.spark.sql.execution.metric.SQLMetric
-class WriteFilesMetricsUpdater(val metrics: Map[String, SQLMetric]) extends MetricsUpdater {
+class WriteFilesMetricsUpdater(override val metrics: Map[String, SQLMetric])
+ extends SharedWriteFilesMetricsUpdater(metrics) {
override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
if (opMetrics != null) {
val operatorMetrics = opMetrics.asInstanceOf[OperatorMetrics]
- metrics("physicalWrittenBytes") += operatorMetrics.physicalWrittenBytes
- metrics("numWrittenFiles") += operatorMetrics.numWrittenFiles
+ updateSharedMetrics(operatorMetrics.physicalWrittenBytes, operatorMetrics.numWrittenFiles)
}
}
}
diff --git a/backends-core/pom.xml b/backends-core/pom.xml
new file mode 100644
index 0000000000..6e3fa13b70
--- /dev/null
+++ b/backends-core/pom.xml
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-parent</artifactId>
+ <version>1.8.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>backends-core</artifactId>
+ <packaging>jar</packaging>
+ <name>Gluten Backends Core</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-substrait</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>${scala.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>net.alchim31.maven</groupId>
+ <artifactId>scala-maven-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.scalastyle</groupId>
+ <artifactId>scalastyle-maven-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>com.diffplug.spotless</groupId>
+ <artifactId>spotless-maven-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>flatten-maven-plugin</artifactId>
+ </plugin>
+ </plugins>
+ <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
+ <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+ </build>
+</project>
diff --git a/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/shared/SharedWriteFilesMetricsUpdater.scala
similarity index 65%
copy from backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
copy to backends-core/src/main/scala/org/apache/gluten/metrics/shared/SharedWriteFilesMetricsUpdater.scala
index 5a04b40433..0576101c9e 100644
--- a/backends-clickhouse/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/metrics/shared/SharedWriteFilesMetricsUpdater.scala
@@ -14,17 +14,19 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.gluten.metrics
+package org.apache.gluten.metrics.shared
+
+import org.apache.gluten.metrics.MetricsUpdater
import org.apache.spark.sql.execution.metric.SQLMetric
-class WriteFilesMetricsUpdater(val metrics: Map[String, SQLMetric]) extends MetricsUpdater {
+abstract class SharedWriteFilesMetricsUpdater(val metrics: Map[String, SQLMetric])
+ extends MetricsUpdater {
- override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
- if (opMetrics != null) {
- val operatorMetrics = opMetrics.asInstanceOf[OperatorMetrics]
- metrics("physicalWrittenBytes") += operatorMetrics.physicalWrittenBytes
- metrics("numWrittenFiles") += operatorMetrics.numWrittenFiles
- }
+ final protected def updateSharedMetrics(
+ physicalWrittenBytes: Long,
+ numWrittenFiles: Long): Unit = {
+ metrics("physicalWrittenBytes") += physicalWrittenBytes
+ metrics("numWrittenFiles") += numWrittenFiles
}
}
diff --git a/backends-velox/pom.xml b/backends-velox/pom.xml
index a8d032a38b..3e81895ea7 100644
--- a/backends-velox/pom.xml
+++ b/backends-velox/pom.xml
@@ -40,6 +40,12 @@
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>backends-core</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala b/backends-velox/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
index 7dc0ca880c..9df6a298d7 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/metrics/WriteFilesMetricsUpdater.scala
@@ -16,17 +16,19 @@
*/
package org.apache.gluten.metrics
+import org.apache.gluten.metrics.shared.SharedWriteFilesMetricsUpdater
+
import org.apache.spark.sql.execution.metric.SQLMetric
-class WriteFilesMetricsUpdater(val metrics: Map[String, SQLMetric]) extends MetricsUpdater {
+class WriteFilesMetricsUpdater(override val metrics: Map[String, SQLMetric])
+ extends SharedWriteFilesMetricsUpdater(metrics) {
override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
if (opMetrics != null) {
val operatorMetrics = opMetrics.asInstanceOf[OperatorMetrics]
- metrics("physicalWrittenBytes") += operatorMetrics.physicalWrittenBytes
+ updateSharedMetrics(operatorMetrics.physicalWrittenBytes, operatorMetrics.numWrittenFiles)
metrics("writeIONanos") += operatorMetrics.writeIOTime
metrics("wallNanos") += operatorMetrics.wallNanos
- metrics("numWrittenFiles") += operatorMetrics.numWrittenFiles
metrics("loadLazyVectorTime") += operatorMetrics.loadLazyVectorTime
}
}
diff --git a/pom.xml b/pom.xml
index d69c773f64..aa8ea0754e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -46,6 +46,7 @@
<modules>
<module>gluten-core</module>
<module>gluten-substrait</module>
+ <module>backends-core</module>
<module>gluten-ui</module>
<module>package</module>
<module>shims</module>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 34/38: [GLUTEN][CORE] Share ColumnarCollectTailExec under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 3bc62436837e0a2d8b8b6aa57526d46853ee53e0
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Wed Jul 1 16:02:51 2026 +0800
[GLUTEN][CORE] Share ColumnarCollectTailExec under backends-core
Move the common CollectTail implementation into backends-core so the velox backend keeps only the backend-specific slicing hook. This reduces duplicated JVM execution code and keeps future backend sync work smaller.
---
.../execution/SharedColumnarCollectTailExec.scala | 18 ++++----
.../gluten/execution/ColumnarCollectTailExec.scala | 53 ++--------------------
2 files changed, 12 insertions(+), 59 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala b/backends-core/src/main/scala/org/apache/gluten/execution/SharedColumnarCollectTailExec.scala
similarity index 82%
copy from backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala
copy to backends-core/src/main/scala/org/apache/gluten/execution/SharedColumnarCollectTailExec.scala
index 7977c9d610..6c4c6437cb 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/execution/SharedColumnarCollectTailExec.scala
@@ -17,7 +17,6 @@
package org.apache.gluten.execution
import org.apache.gluten.columnarbatch.ColumnarBatches
-import org.apache.gluten.columnarbatch.VeloxColumnarBatches
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.vectorized.ColumnarBatch
@@ -25,10 +24,14 @@ import org.apache.spark.sql.vectorized.ColumnarBatch
import scala.collection.mutable
import scala.util.control.Breaks._
-case class ColumnarCollectTailExec(
- limit: Int,
- child: SparkPlan
-) extends ColumnarCollectTailBaseExec(limit, child) {
+/**
+ * Shared collect-tail implementation for backends that can retain incoming columnar batches and
+ * slice the first overflow batch using a backend-specific primitive.
+ */
+abstract class SharedColumnarCollectTailExec(limit: Int, child: SparkPlan)
+ extends ColumnarCollectTailBaseExec(limit, child) {
+
+ protected def sliceBatch(batch: ColumnarBatch, offset: Int, length: Int): ColumnarBatch
override protected def collectTailRows(
partitionIter: Iterator[ColumnarBatch],
@@ -68,14 +71,11 @@ case class ColumnarCollectTailExec(
if (overflow > 0) {
val first = tailQueue.remove(0)
val keep = first.numRows() - overflow
- val sliced = VeloxColumnarBatches.slice(first, overflow.toInt, keep.toInt)
+ val sliced = sliceBatch(first, overflow.toInt, keep.toInt)
tailQueue.prepend(sliced)
first.close()
}
tailQueue.iterator
}
-
- override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =
- copy(child = newChild)
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala b/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala
index 7977c9d610..cbc3c42a07 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarCollectTailExec.scala
@@ -16,65 +16,18 @@
*/
package org.apache.gluten.execution
-import org.apache.gluten.columnarbatch.ColumnarBatches
import org.apache.gluten.columnarbatch.VeloxColumnarBatches
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.vectorized.ColumnarBatch
-import scala.collection.mutable
-import scala.util.control.Breaks._
-
case class ColumnarCollectTailExec(
limit: Int,
child: SparkPlan
-) extends ColumnarCollectTailBaseExec(limit, child) {
-
- override protected def collectTailRows(
- partitionIter: Iterator[ColumnarBatch],
- limit: Int
- ): Iterator[ColumnarBatch] = {
- if (!partitionIter.hasNext || limit <= 0) {
- return Iterator.empty
- }
-
- val tailQueue = new mutable.ListBuffer[ColumnarBatch]()
- var totalRowsInTail = 0L
-
- while (partitionIter.hasNext) {
- val batch = partitionIter.next()
- val batchRows = batch.numRows()
- ColumnarBatches.retain(batch)
- tailQueue += batch
- totalRowsInTail += batchRows
-
- breakable {
- while (tailQueue.nonEmpty) {
- val front = tailQueue.head
- val frontRows = front.numRows()
-
- if (totalRowsInTail - frontRows >= limit) {
- val dropped = tailQueue.remove(0)
- dropped.close()
- totalRowsInTail -= frontRows
- } else {
- break
- }
- }
- }
- }
-
- val overflow = totalRowsInTail - limit
- if (overflow > 0) {
- val first = tailQueue.remove(0)
- val keep = first.numRows() - overflow
- val sliced = VeloxColumnarBatches.slice(first, overflow.toInt, keep.toInt)
- tailQueue.prepend(sliced)
- first.close()
- }
+) extends SharedColumnarCollectTailExec(limit, child) {
- tailQueue.iterator
- }
+ override protected def sliceBatch(batch: ColumnarBatch, offset: Int, length: Int): ColumnarBatch =
+ VeloxColumnarBatches.slice(batch, offset, length)
override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =
copy(child = newChild)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 15/38: [GLUTEN][CORE] Share array-to-string cast rewrite rule
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 50bd8f717f0a393830140a32b66d59d102eb6461
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:07:21 2026 +0800
[GLUTEN][CORE] Share array-to-string cast rewrite rule
Move the reusable cast(array as string) rewrite logic into backends-core and keep the Velox rule as a thin config-gated wrapper so later backends can reuse the same optimizer rule body.
---
.../extension/SharedRewriteCastFromArrayRule.scala | 15 ++--------
.../gluten/extension/RewriteCastFromArray.scala | 33 ++--------------------
2 files changed, 4 insertions(+), 44 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala b/backends-core/src/main/scala/org/apache/gluten/extension/SharedRewriteCastFromArrayRule.scala
similarity index 78%
copy from backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala
copy to backends-core/src/main/scala/org/apache/gluten/extension/SharedRewriteCastFromArrayRule.scala
index 4821418852..e3cd00770a 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/extension/SharedRewriteCastFromArrayRule.scala
@@ -16,9 +16,6 @@
*/
package org.apache.gluten.extension
-import org.apache.gluten.config.VeloxConfig
-
-import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.expressions.{ArrayJoin, Cast, Concat, Literal}
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.rules.Rule
@@ -26,17 +23,9 @@ import org.apache.spark.sql.catalyst.trees.TreePattern.CAST
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{ArrayType, StringType}
-/**
- * Velox does not support cast Array to String. Before velox support, temporarily add this rule to
- * replace `cast(array as String)` with `concat('[', array_join(array, ', ', 'null'), ']')` to
- * support offload.
- */
-case class RewriteCastFromArray(spark: SparkSession) extends Rule[LogicalPlan] {
+class SharedRewriteCastFromArrayRule(enabled: Boolean) extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan = {
- if (
- !VeloxConfig.get.enableRewriteCastArrayToString ||
- SQLConf.get.getConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING)
- ) {
+ if (!enabled || SQLConf.get.getConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING)) {
return plan
}
plan.transformUpWithPruning(_.containsPattern(CAST)) {
diff --git a/backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala b/backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala
index 4821418852..f0251abf02 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/extension/RewriteCastFromArray.scala
@@ -19,40 +19,11 @@ package org.apache.gluten.extension
import org.apache.gluten.config.VeloxConfig
import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.expressions.{ArrayJoin, Cast, Concat, Literal}
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.trees.TreePattern.CAST
-import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{ArrayType, StringType}
/**
* Velox does not support cast Array to String. Before velox support, temporarily add this rule to
* replace `cast(array as String)` with `concat('[', array_join(array, ', ', 'null'), ']')` to
* support offload.
*/
-case class RewriteCastFromArray(spark: SparkSession) extends Rule[LogicalPlan] {
- override def apply(plan: LogicalPlan): LogicalPlan = {
- if (
- !VeloxConfig.get.enableRewriteCastArrayToString ||
- SQLConf.get.getConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING)
- ) {
- return plan
- }
- plan.transformUpWithPruning(_.containsPattern(CAST)) {
- case p =>
- p.transformExpressionsUpWithPruning(_.containsPattern(CAST)) {
- case Cast(child, StringType, timeZoneId, evalMode)
- if child.dataType.isInstanceOf[ArrayType] =>
- val joinChild = child.dataType.asInstanceOf[ArrayType].elementType match {
- case StringType =>
- child
- case _ =>
- Cast(child, ArrayType(StringType), timeZoneId, evalMode)
- }
- val arrayJoin = ArrayJoin(joinChild, Literal(", "), Some(Literal("null")))
- Concat(Seq(Literal("["), arrayJoin, Literal("]")))
- }
- }
- }
-}
+case class RewriteCastFromArray(spark: SparkSession)
+ extends SharedRewriteCastFromArrayRule(VeloxConfig.get.enableRewriteCastArrayToString)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 05/38: [GLUTEN][VL] Support format_number function in Velox backend
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit f9aa6e0ffa774b2f05acaeb6fad47fbd627b3b17
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Fri Jun 26 18:05:19 2026 +0800
[GLUTEN][VL] Support format_number function in Velox backend
Register the Spark format_number expression mapping (and the
FORMAT_NUMBER expression name) so it can be offloaded to Velox, which
natively supports format_number. Add a UT covering column inputs
(non constant-folding) and update the scalar function support doc.
Generated-by: TraeCli openrouter-3o
---
.../gluten/functions/ScalarFunctionsValidateSuite.scala | 11 +++++++++++
docs/velox-backend-scalar-function-support.md | 4 ++--
.../org/apache/gluten/expression/ExpressionMappings.scala | 1 +
.../scala/org/apache/gluten/expression/ExpressionNames.scala | 1 +
4 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala b/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
index 38db4e9b3a..7e9f433142 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
+++ b/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
@@ -551,6 +551,17 @@ abstract class ScalarFunctionsValidateSuite extends FunctionsValidateSuite {
}
}
+ test("format_number function with column inputs offloaded to native") {
+ runQueryAndCompare("""SELECT format_number(l_extendedprice, 2), l_orderkey
+ | from lineitem limit 100""".stripMargin) {
+ checkGlutenPlan[ProjectExecTransformer]
+ }
+ runQueryAndCompare("""SELECT format_number(l_quantity, l_linenumber), l_orderkey
+ | from lineitem limit 100""".stripMargin) {
+ checkGlutenPlan[ProjectExecTransformer]
+ }
+ }
+
test("map_from_arrays optimized by Spark constant folding") {
withSQLConf(("spark.sql.optimizer.excludedRules", "")) {
runQueryAndCompare("""SELECT map_from_arrays(sequence(1, 5),sequence(1, 5)), l_orderkey
diff --git a/docs/velox-backend-scalar-function-support.md b/docs/velox-backend-scalar-function-support.md
index 1da7586a6d..617c06f9b5 100644
--- a/docs/velox-backend-scalar-function-support.md
+++ b/docs/velox-backend-scalar-function-support.md
@@ -1,6 +1,6 @@
# Scalar Functions Support Status
-**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 243 functions and partially supports 25 functions.**
+**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 244 functions and partially supports 25 functions.**
## Array Functions
@@ -369,7 +369,7 @@
| encode | Encode | | |
| endswith | EndsWithExpressionBuilder | PS | BinaryType unsupported |
| find_in_set | FindInSet | S | |
-| format_number | FormatNumber | | |
+| format_number | FormatNumber | S | |
| format_string | FormatString | | |
| initcap | InitCap | S | |
| instr | StringInstr | S | |
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala b/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala
index d0ed9347b6..5193f31d66 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala
@@ -107,6 +107,7 @@ object ExpressionMappings {
Sig[UnBase64](UNBASE64),
Sig[Base64](BASE64),
Sig[FormatString](FORMAT_STRING),
+ Sig[FormatNumber](FORMAT_NUMBER),
// URL functions
Sig[ParseUrl](PARSE_URL),
diff --git a/shims/common/src/main/scala/org/apache/gluten/expression/ExpressionNames.scala b/shims/common/src/main/scala/org/apache/gluten/expression/ExpressionNames.scala
index ca05f0ade1..03c41ebc49 100644
--- a/shims/common/src/main/scala/org/apache/gluten/expression/ExpressionNames.scala
+++ b/shims/common/src/main/scala/org/apache/gluten/expression/ExpressionNames.scala
@@ -142,6 +142,7 @@ object ExpressionNames {
final val BASE64 = "base64"
final val MASK = "mask"
final val FORMAT_STRING = "format_string"
+ final val FORMAT_NUMBER = "format_number"
final val LUHN_CHECK = "luhn_check"
final val TO_PRETTY_STRING = "to_pretty_string"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 38/38: add bolt backend in gluten
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit fe46c376c54f1cf10fefd69550187d2c48a50ed0
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Fri Oct 10 18:45:25 2025 +0800
add bolt backend in gluten
---
.github/skills/extract-cpp-common/SKILL.md | 87 +
.github/skills/extract-jvm-common/SKILL.md | 118 +
.github/skills/resolve-main-conflict/SKILL.md | 104 +
.gitignore | 4 +
.idea/vcs.xml | 23 +-
Makefile | 226 ++
README.md | 54 +
.../ColumnarTableCacheBenchmark-results.txt | 23 +
backends-bolt/pom.xml | 540 +++
.../CelebornPartitionWriterJniWrapper.java | 47 +
...celeborn.CelebornColumnarBatchSerializerFactory | 1 +
...le.gluten.celeborn.CelebornShuffleWriterFactory | 1 +
.../BoltCelebornColumnarBatchSerializer.scala | 322 ++
...oltCelebornColumnarBatchSerializerFactory.scala | 13 +-
.../BoltCelebornColumnarShuffleWriter.scala | 251 ++
.../BoltCelebornColumnarShuffleWriterFactory.scala | 33 +-
.../org.apache.gluten.component.BoltDeltaComponent | 0
.../gluten/component/BoltDeltaComponent.scala | 60 +
.../apache/gluten/execution/BoltDeltaSuite.scala | 11 +-
.../gluten/execution/BoltTPCHDeltaSuite.scala | 58 +
.../apache/spark/sql/delta/DeleteSQLSuite.scala | 149 +
.../apache/spark/sql/delta/DeleteSuiteBase.scala | 565 ++++
.../spark/sql/delta/DeletionVectorsTestUtils.scala | 367 ++
.../sql/delta/DeltaColumnMappingTestUtils.scala | 487 +++
.../apache/spark/sql/delta/DeltaTestUtils.scala | 635 ++++
.../test/DeltaColumnMappingSelectedTestMixin.scala | 76 +
.../sql/delta/test/DeltaExcludedTestMixin.scala | 31 +-
.../spark/sql/delta/test/DeltaSQLCommandTest.scala | 52 +
.../spark/sql/delta/test/DeltaSQLTestUtils.scala | 79 +
.../spark/sql/delta/test/DeltaTestImplicits.scala | 204 ++
...DeltaExcludedBySparkVersionTestMixinShims.scala | 45 +
.../org.apache.gluten.component.BoltHudiComponent | 0
.../gluten/component/BoltHudiComponent.scala | 52 +
.../apache/gluten/execution/BoltHudiSuite.scala | 11 +-
.../gluten/execution/BoltTPCHHudiSuite.scala | 60 +
.../execution/TestStoragePartitionedJoins.java | 665 ++++
.../execution/TestTPCHStoragePartitionedJoins.java | 247 ++
.../extensions/TestGlutenCopyOnWriteDelete.java | 63 +
.../extensions/TestGlutenMergeOnReadDelete.java | 63 +
.../extensions/TestGlutenMergeOnReadMerge.java | 145 +
.../extensions/TestGlutenMergeOnReadUpdate.java | 63 +
...toragePartitionedJoinsInRowLevelOperations.java | 19 +-
.../TestGlutenSystemFunctionPushDownDQL.java | 18 +-
...SystemFunctionPushDownInRowLevelOperations.java | 17 +-
.../apache/gluten/source/TestDataFrameWrites.java | 403 +++
.../org/apache/gluten/source/TestFilteredScan.java | 726 ++++
.../gluten/source/TestForwardCompatibility.java | 213 ++
.../gluten/source/TestGlutenDataFrameWriterV2.java | 11 +-
.../TestGlutenDataFrameWriterV2Coercion.java | 16 +-
.../gluten/source/TestGlutenDataSourceOptions.java | 11 +-
.../source/TestGlutenIcebergSourceHiveTables.java | 12 +-
.../source/TestGlutenIdentityPartitionData.java | 17 +-
.../source/TestGlutenPositionDeletesTable.java | 19 +-
.../gluten/source/TestGlutenRuntimeFiltering.java | 16 +-
.../source/TestGlutenSparkMetadataColumns.java | 17 +-
.../gluten/source/TestGlutenSparkStagedScan.java | 18 +-
.../org/apache/gluten/source/TestIcebergSpark.java | 252 ++
.../org/apache/gluten/source/TestParquetScan.java | 336 ++
.../apache/gluten/source/TestPartitionPruning.java | 479 +++
.../apache/gluten/source/TestPartitionValues.java | 505 +++
.../apache/gluten/source/TestSparkDataFile.java | 312 ++
.../source/TestSparkReaderWithBloomFilter.java | 354 ++
.../gluten/source/TestStructuredStreaming.java | 298 ++
.../gluten/source/TestTimestampWithoutZone.java | 231 ++
.../org/apache/gluten/sql/TestFilterPushDown.java | 604 ++++
.../gluten/sql/TestGlutenAggregatePushDown.java | 65 +
.../apache/gluten/sql/TestGlutenDeleteFrom.java | 18 +-
.../sql/TestGlutenPartitionedWritesAsSelect.java | 11 +-
.../sql/TestGlutenPartitionedWritesToBranch.java | 18 +-
.../TestGlutenPartitionedWritesToWapBranch.java | 18 +-
.../org/apache/gluten/sql/TestGlutenSelect.java | 17 +-
.../gluten/sql/TestGlutenTimestampWithoutZone.java | 18 +-
.../org/apache/iceberg/spark/SparkTestBase.java | 305 ++
.../spark/extensions/SparkExtensionsTestBase.java | 73 +
.../spark/source/GlutenSparkScanBuilder.java | 48 +
.../spark/source/TestIcebergSourceTablesBase.java | 2282 +++++++++++++
.../spark/source/TestSparkReadProjection.java | 267 ++
.../spark/source/TestSparkReaderDeletes.java | 686 ++++
.../main/java/org/apache/gluten/TestConfUtil.java | 26 +-
...rg.apache.gluten.component.BoltIcebergComponent | 0
.../gluten/component/BoltIcebergComponent.scala | 20 +-
.../write/IcebergColumnarBatchDataWriter.scala | 89 +
.../connector/write/IcebergDataWriteFactory.scala | 129 +
.../execution/AbstractIcebergWriteExec.scala | 87 +
.../execution/BoltIcebergAppendDataExec.scala | 29 +-
.../BoltIcebergOverwriteByExpressionExec.scala | 32 +-
...BoltIcebergOverwritePartitionsDynamicExec.scala | 32 +-
.../execution/BoltIcebergReplaceDataExec.scala | 29 +-
.../BoltIcebergWriteToDataSourceV2Exec.scala | 84 +
.../gluten/execution/IcebergWriteJniWrapper.java | 51 +
.../gluten/extension/OffloadIcebergWrite.scala | 89 +
.../gluten/TestIcebergNestedFieldVisitor.java | 125 +
.../org/apache/iceberg/spark/data/RandomData.java | 364 ++
.../org/apache/iceberg/spark/data/TestHelpers.java | 807 +++++
.../apache/iceberg/spark/source/LogMessage.java | 118 +
.../apache/gluten/execution/BoltIcebergSuite.scala | 11 +-
.../execution/enhanced/BoltIcebergSuite.scala | 286 ++
.../org/apache/spark/sql/gluten/TestUtils.scala | 46 +
...org.apache.gluten.component.BoltPaimonComponent | 0
.../gluten/component/BoltPaimonComponent.scala | 125 +
.../apache/gluten/execution/BoltPaimonSuite.scala | 2394 +++++++++++++
.../UnifflePartitionWriterJniWrapper.java | 36 +-
.../gluten/uniffle/UniffleShuffleManager.java | 87 +
.../writer/BoltUniffleColumnarShuffleWriter.java | 301 ++
.../spark/shuffle/writer/PartitionPusher.scala | 17 +-
.../columnarbatch/BoltColumnarBatchJniWrapper.java | 35 +-
.../gluten/columnarbatch/BoltColumnarBatches.java | 162 +
.../cudf/BoltCudfPlanValidatorJniWrapper.java | 14 +-
.../datasource/BoltDataSourceJniWrapper.java | 64 +
.../org/apache/gluten/jni/BoltJniLibLoader.java | 454 +++
.../apache/gluten/metrics/BatchWriteMetrics.java | 59 +
.../apache/gluten/monitor/BoltMemoryProfiler.java | 42 +
.../shuffle/BoltShuffleReaderJniWrapper.java | 46 +
.../gluten/shuffle/BoltShuffleReaderMetrics.java | 28 +-
.../shuffle/BoltShuffleWriterJniWrapper.java | 106 +
.../org/apache/gluten/shuffle/BoltSplitResult.java | 160 +
.../org/apache/gluten/utils/BoltBatchResizer.java | 42 +
.../gluten/utils/BoltBatchResizerJniWrapper.java | 30 +-
.../org/apache/gluten/utils/BoltBloomFilter.java | 189 ++
.../gluten/utils/BoltBloomFilterJniWrapper.java | 51 +
.../utils/BoltFileSystemValidationJniWrapper.java | 12 +-
.../org.apache.gluten.backendsapi.bolt.BoltBackend | 0
.../org.apache.gluten.spi.SharedLibraryLoader | 20 +-
.../gluten/proto/PaimonTableEnhancement.proto | 11 +
.../apache/gluten/proto/shuffle_reader_info.proto | 18 +
.../apache/gluten/proto/shuffle_writer_info.proto | 89 +
.../gluten/backendsapi/bolt/BoltBackend.scala | 581 ++++
.../gluten/backendsapi/bolt/BoltBatchType.scala | 23 +-
.../backendsapi/bolt/BoltCarrierRowType.scala | 16 +-
.../gluten/backendsapi/bolt/BoltIteratorApi.scala | 430 +++
.../gluten/backendsapi/bolt/BoltListenerApi.scala | 306 ++
.../gluten/backendsapi/bolt/BoltMetricsApi.scala | 761 +++++
.../gluten/backendsapi/bolt/BoltRuleApi.scala | 147 +
.../backendsapi/bolt/BoltSparkPlanExecApi.scala | 1388 ++++++++
.../backendsapi/bolt/BoltTransformerApi.scala | 125 +
.../gluten/backendsapi/bolt/BoltValidatorApi.scala | 118 +
.../org/apache/gluten/config/BoltConfig.scala | 920 +++++
.../gluten/datasource/BoltDataSourceUtil.scala | 55 +
.../ArrowColumnarToBoltColumnarExec.scala | 22 +-
.../execution/BoltBroadcastBuildSideRDD.scala | 28 +-
...oltBroadcastNestedLoopJoinExecTransformer.scala | 77 +
.../execution/BoltColumnarToCarrierRowExec.scala | 27 +-
.../gluten/execution/BoltColumnarToRowExec.scala | 209 ++
.../gluten/execution/BoltResizeBatchesExec.scala | 63 +
.../execution/ColumnarCollectLimitExec.scala | 100 +
.../gluten/execution/ColumnarCollectTailExec.scala | 23 +-
.../execution/ColumnarPartialGenerateExec.scala | 402 +++
.../execution/ColumnarPartialProjectExec.scala | 357 ++
.../gluten/execution/ColumnarRangeExec.scala | 129 +
.../gluten/execution/FilterExecTransformer.scala | 17 +-
.../gluten/execution/GenerateExecTransformer.scala | 372 ++
.../execution/HashAggregateExecTransformer.scala | 714 ++++
.../gluten/execution/HashJoinExecTransformer.scala | 133 +
.../gluten/execution/RowToBoltColumnarExec.scala | 73 +
.../expression/BoltBloomFilterMightContain.scala | 148 +
.../apache/gluten/expression/DummyExpression.scala | 77 +
.../gluten/expression/ExpressionRestrictions.scala | 130 +
.../gluten/expression/ExpressionTransformer.scala | 126 +
.../aggregate/BoltBloomFilterAggregate.scala | 171 +
.../gluten/expression/aggregate/BoltCollect.scala | 71 +
.../gluten/expression/aggregate/HLLAdapter.scala | 116 +
...AppendBatchResizeForShuffleInputAndOutput.scala | 102 +
.../BloomFilterMightContainJointRewriteRule.scala | 83 +
.../gluten/extension/CollectRewriteRule.scala | 81 +
.../gluten/extension/CudfNodeValidationRule.scala | 47 +
.../extension/FlushableHashAggregateRule.scala | 141 +
.../apache/gluten/extension/HLLRewriteRule.scala | 66 +
.../HashAggregateIgnoreNullKeysRule.scala | 28 +-
.../gluten/extension/PartialGenerateRule.scala | 63 +
.../gluten/extension/PartialProjectRule.scala | 62 +
.../gluten/extension/PullOutDuplicateProject.scala | 137 +
.../PushdownProjectExecBeforeGeneratorRule.scala | 112 +
.../RemoveProjectExecBeforeGeneratorRule.scala | 69 +
.../gluten/extension/RewriteCastFromArray.scala | 19 +-
.../gluten/extension/RewriteUnboundedWindow.scala | 83 +
.../gluten/metrics/BatchScanMetricsUpdater.scala | 62 +
.../metrics/FileSourceScanMetricsUpdater.scala | 91 +
.../metrics/HashAggregateMetricsUpdater.scala | 21 +-
.../metrics/HiveTableScanMetricsUpdater.scala | 86 +
.../metrics/InputIteratorMetricsUpdater.scala | 14 +-
.../apache/gluten/metrics/JoinMetricsUpdater.scala | 213 ++
.../org/apache/gluten/metrics/MetricsUtil.scala | 377 +++
.../apache/gluten/spi/SharedLibraryLoader.scala | 40 +-
.../gluten/spi/SharedLibraryLoaderCentos7.scala | 50 +
.../gluten/spi/SharedLibraryLoaderCentos8.scala | 53 +
.../gluten/spi/SharedLibraryLoaderCentos9.scala | 49 +
.../gluten/spi/SharedLibraryLoaderDebian11.scala | 54 +
.../gluten/spi/SharedLibraryLoaderDebian12.scala | 60 +
.../gluten/spi/SharedLibraryLoaderMacOS.scala | 20 +-
.../spi/SharedLibraryLoaderOpenEuler2403.scala | 50 +
.../gluten/spi/SharedLibraryLoaderUbuntu2004.scala | 69 +
.../gluten/spi/SharedLibraryLoaderUbuntu2204.scala | 54 +
.../apache/gluten/utils/BoltIntermediateData.scala | 209 ++
.../apache/gluten/utils/ParquetMetadataUtils.scala | 170 +
.../gluten/utils/SharedLibraryLoaderUtils.scala | 100 +
.../vectorized/ColumnarBatchSerializer.scala | 290 ++
.../SettableColumnarBatchSerializer.scala | 45 +
.../api/python/ColumnarArrowEvalPythonExec.scala | 474 +++
.../spark/metrics/TaskStatsAccumulator.scala | 60 +
.../spark/shuffle/ColumnarShuffleReader.scala | 217 ++
.../spark/shuffle/ColumnarShuffleWriter.scala | 415 +++
.../apache/spark/shuffle/utils/ShuffleUtil.scala | 56 +
.../spark/sql/execution/BaseArrowScanExec.scala | 19 +-
.../sql/execution/BoltColumnarWriteFilesExec.scala | 294 ++
.../spark/sql/execution/BroadcastModeUtils.scala | 134 +
.../spark/sql/execution/BroadcastUtils.scala | 216 ++
.../sql/execution/ColumnarBuildSideRelation.scala | 239 ++
.../execution/ColumnarCachedBatchSerializer.scala | 847 +++++
.../execution/SparkWriteFilesCommitProtocol.scala | 134 +
.../datasources/BoltParquetBatchWriter.scala | 114 +
.../execution/datasources/ParquetEncryption.scala | 80 +
.../datasources/bolt/BoltBlockStripes.java | 76 +
.../datasources/bolt/BoltFormatWriterInjects.scala | 124 +
.../bolt/BoltParquetWriterInjects.scala | 73 +
.../execution/unsafe/UnsafeBytesBufferArray.scala | 138 +
.../unsafe/UnsafeColumnarBuildSideRelation.scala | 369 ++
.../spark/sql/execution/utils/ExecUtil.scala | 257 ++
.../apache/spark/sql/expression/UDFResolver.scala | 480 +++
.../spark/sql/hive/BoltHiveUDFTransformer.scala | 54 +
.../gluten/backendsapi/BoltListenerApiTest.java | 32 +-
.../gluten/columnarbatch/ColumnarBatchTest.java | 253 ++
.../org/apache/gluten/fs/OnHeapFileSystemTest.java | 58 +
.../apache/gluten/test/BoltBackendTestBase.java | 27 +-
.../org/apache/gluten/test/MockBoltBackend.java | 78 +
.../apache/gluten/utils/BoltBloomFilterTest.java | 211 ++
.../gluten/vectorized/ArrowColumnVectorTest.java | 50 +
.../org.apache.gluten.spi.SharedLibraryLoader | 12 +-
.../src/test/resources/log4j2.properties | 31 +-
.../tpch-approved-plan/v1-bhj-ras/spark32/1.txt | 162 +
.../tpch-approved-plan/v1-bhj-ras/spark32/10.txt | 368 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/11.txt | 320 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/12.txt | 236 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/13.txt | 297 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/14.txt | 195 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/15.txt | 246 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/16.txt | 323 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/17.txt | 203 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/18.txt | 480 +++
.../tpch-approved-plan/v1-bhj-ras/spark32/19.txt | 190 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/20.txt | 585 ++++
.../tpch-approved-plan/v1-bhj-ras/spark32/21.txt | 499 +++
.../tpch-approved-plan/v1-bhj-ras/spark32/22.txt | 214 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/3.txt | 294 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/4.txt | 246 ++
.../tpch-approved-plan/v1-bhj-ras/spark32/5.txt | 542 +++
.../tpch-approved-plan/v1-bhj-ras/spark32/6.txt | 116 +
.../tpch-approved-plan/v1-bhj-ras/spark32/7.txt | 504 +++
.../tpch-approved-plan/v1-bhj-ras/spark32/8.txt | 695 ++++
.../tpch-approved-plan/v1-bhj-ras/spark32/9.txt | 532 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/1.txt | 162 +
.../tpch-approved-plan/v1-bhj-ras/spark33/10.txt | 368 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/11.txt | 551 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/12.txt | 236 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/13.txt | 297 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/14.txt | 195 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/15.txt | 388 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/16.txt | 323 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/17.txt | 203 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/18.txt | 480 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/19.txt | 190 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/20.txt | 574 ++++
.../tpch-approved-plan/v1-bhj-ras/spark33/21.txt | 494 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/22.txt | 354 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/3.txt | 294 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/4.txt | 246 ++
.../tpch-approved-plan/v1-bhj-ras/spark33/5.txt | 542 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/6.txt | 116 +
.../tpch-approved-plan/v1-bhj-ras/spark33/7.txt | 504 +++
.../tpch-approved-plan/v1-bhj-ras/spark33/8.txt | 695 ++++
.../tpch-approved-plan/v1-bhj-ras/spark33/9.txt | 532 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/1.txt | 162 +
.../tpch-approved-plan/v1-bhj-ras/spark34/10.txt | 374 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/11.txt | 559 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/12.txt | 238 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/13.txt | 299 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/14.txt | 197 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/15.txt | 390 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/16.txt | 326 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/17.txt | 205 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/18.txt | 488 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/19.txt | 192 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/20.txt | 533 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/21.txt | 504 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/22.txt | 356 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/3.txt | 298 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/4.txt | 248 ++
.../tpch-approved-plan/v1-bhj-ras/spark34/5.txt | 552 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/6.txt | 116 +
.../tpch-approved-plan/v1-bhj-ras/spark34/7.txt | 514 +++
.../tpch-approved-plan/v1-bhj-ras/spark34/8.txt | 709 ++++
.../tpch-approved-plan/v1-bhj-ras/spark34/9.txt | 542 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/1.txt | 162 +
.../tpch-approved-plan/v1-bhj-ras/spark35/10.txt | 374 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/11.txt | 559 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/12.txt | 238 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/13.txt | 299 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/14.txt | 197 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/15.txt | 390 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/16.txt | 326 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/17.txt | 205 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/18.txt | 488 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/19.txt | 192 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/20.txt | 533 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/21.txt | 504 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/22.txt | 356 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/3.txt | 298 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/4.txt | 248 ++
.../tpch-approved-plan/v1-bhj-ras/spark35/5.txt | 552 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/6.txt | 116 +
.../tpch-approved-plan/v1-bhj-ras/spark35/7.txt | 514 +++
.../tpch-approved-plan/v1-bhj-ras/spark35/8.txt | 709 ++++
.../tpch-approved-plan/v1-bhj-ras/spark35/9.txt | 542 +++
.../tpch-approved-plan/v1-bhj/spark32/1.txt | 162 +
.../tpch-approved-plan/v1-bhj/spark32/10.txt | 368 ++
.../tpch-approved-plan/v1-bhj/spark32/11.txt | 320 ++
.../tpch-approved-plan/v1-bhj/spark32/12.txt | 236 ++
.../tpch-approved-plan/v1-bhj/spark32/13.txt | 297 ++
.../tpch-approved-plan/v1-bhj/spark32/14.txt | 195 ++
.../tpch-approved-plan/v1-bhj/spark32/15.txt | 246 ++
.../tpch-approved-plan/v1-bhj/spark32/16.txt | 323 ++
.../tpch-approved-plan/v1-bhj/spark32/17.txt | 203 ++
.../tpch-approved-plan/v1-bhj/spark32/18.txt | 480 +++
.../tpch-approved-plan/v1-bhj/spark32/19.txt | 190 ++
.../tpch-approved-plan/v1-bhj/spark32/20.txt | 585 ++++
.../tpch-approved-plan/v1-bhj/spark32/21.txt | 499 +++
.../tpch-approved-plan/v1-bhj/spark32/22.txt | 214 ++
.../tpch-approved-plan/v1-bhj/spark32/3.txt | 294 ++
.../tpch-approved-plan/v1-bhj/spark32/4.txt | 246 ++
.../tpch-approved-plan/v1-bhj/spark32/5.txt | 542 +++
.../tpch-approved-plan/v1-bhj/spark32/6.txt | 116 +
.../tpch-approved-plan/v1-bhj/spark32/7.txt | 504 +++
.../tpch-approved-plan/v1-bhj/spark32/8.txt | 695 ++++
.../tpch-approved-plan/v1-bhj/spark32/9.txt | 532 +++
.../tpch-approved-plan/v1-bhj/spark33/1.txt | 162 +
.../tpch-approved-plan/v1-bhj/spark33/10.txt | 368 ++
.../tpch-approved-plan/v1-bhj/spark33/11.txt | 551 +++
.../tpch-approved-plan/v1-bhj/spark33/12.txt | 236 ++
.../tpch-approved-plan/v1-bhj/spark33/13.txt | 297 ++
.../tpch-approved-plan/v1-bhj/spark33/14.txt | 195 ++
.../tpch-approved-plan/v1-bhj/spark33/15.txt | 388 +++
.../tpch-approved-plan/v1-bhj/spark33/16.txt | 323 ++
.../tpch-approved-plan/v1-bhj/spark33/17.txt | 203 ++
.../tpch-approved-plan/v1-bhj/spark33/18.txt | 480 +++
.../tpch-approved-plan/v1-bhj/spark33/19.txt | 190 ++
.../tpch-approved-plan/v1-bhj/spark33/20.txt | 574 ++++
.../tpch-approved-plan/v1-bhj/spark33/21.txt | 494 +++
.../tpch-approved-plan/v1-bhj/spark33/22.txt | 354 ++
.../tpch-approved-plan/v1-bhj/spark33/3.txt | 294 ++
.../tpch-approved-plan/v1-bhj/spark33/4.txt | 246 ++
.../tpch-approved-plan/v1-bhj/spark33/5.txt | 542 +++
.../tpch-approved-plan/v1-bhj/spark33/6.txt | 116 +
.../tpch-approved-plan/v1-bhj/spark33/7.txt | 504 +++
.../tpch-approved-plan/v1-bhj/spark33/8.txt | 695 ++++
.../tpch-approved-plan/v1-bhj/spark33/9.txt | 532 +++
.../tpch-approved-plan/v1-bhj/spark34/1.txt | 162 +
.../tpch-approved-plan/v1-bhj/spark34/10.txt | 374 +++
.../tpch-approved-plan/v1-bhj/spark34/11.txt | 559 +++
.../tpch-approved-plan/v1-bhj/spark34/12.txt | 238 ++
.../tpch-approved-plan/v1-bhj/spark34/13.txt | 299 ++
.../tpch-approved-plan/v1-bhj/spark34/14.txt | 197 ++
.../tpch-approved-plan/v1-bhj/spark34/15.txt | 390 +++
.../tpch-approved-plan/v1-bhj/spark34/16.txt | 326 ++
.../tpch-approved-plan/v1-bhj/spark34/17.txt | 205 ++
.../tpch-approved-plan/v1-bhj/spark34/18.txt | 488 +++
.../tpch-approved-plan/v1-bhj/spark34/19.txt | 192 ++
.../tpch-approved-plan/v1-bhj/spark34/20.txt | 533 +++
.../tpch-approved-plan/v1-bhj/spark34/21.txt | 504 +++
.../tpch-approved-plan/v1-bhj/spark34/22.txt | 356 ++
.../tpch-approved-plan/v1-bhj/spark34/3.txt | 298 ++
.../tpch-approved-plan/v1-bhj/spark34/4.txt | 248 ++
.../tpch-approved-plan/v1-bhj/spark34/5.txt | 552 +++
.../tpch-approved-plan/v1-bhj/spark34/6.txt | 116 +
.../tpch-approved-plan/v1-bhj/spark34/7.txt | 514 +++
.../tpch-approved-plan/v1-bhj/spark34/8.txt | 709 ++++
.../tpch-approved-plan/v1-bhj/spark34/9.txt | 542 +++
.../tpch-approved-plan/v1-bhj/spark35/1.txt | 162 +
.../tpch-approved-plan/v1-bhj/spark35/10.txt | 374 +++
.../tpch-approved-plan/v1-bhj/spark35/11.txt | 559 +++
.../tpch-approved-plan/v1-bhj/spark35/12.txt | 238 ++
.../tpch-approved-plan/v1-bhj/spark35/13.txt | 299 ++
.../tpch-approved-plan/v1-bhj/spark35/14.txt | 197 ++
.../tpch-approved-plan/v1-bhj/spark35/15.txt | 390 +++
.../tpch-approved-plan/v1-bhj/spark35/16.txt | 326 ++
.../tpch-approved-plan/v1-bhj/spark35/17.txt | 205 ++
.../tpch-approved-plan/v1-bhj/spark35/18.txt | 488 +++
.../tpch-approved-plan/v1-bhj/spark35/19.txt | 192 ++
.../tpch-approved-plan/v1-bhj/spark35/20.txt | 533 +++
.../tpch-approved-plan/v1-bhj/spark35/21.txt | 504 +++
.../tpch-approved-plan/v1-bhj/spark35/22.txt | 356 ++
.../tpch-approved-plan/v1-bhj/spark35/3.txt | 298 ++
.../tpch-approved-plan/v1-bhj/spark35/4.txt | 248 ++
.../tpch-approved-plan/v1-bhj/spark35/5.txt | 552 +++
.../tpch-approved-plan/v1-bhj/spark35/6.txt | 116 +
.../tpch-approved-plan/v1-bhj/spark35/7.txt | 514 +++
.../tpch-approved-plan/v1-bhj/spark35/8.txt | 709 ++++
.../tpch-approved-plan/v1-bhj/spark35/9.txt | 542 +++
.../tpch-approved-plan/v1-ras/spark32/1.txt | 162 +
.../tpch-approved-plan/v1-ras/spark32/10.txt | 516 +++
.../tpch-approved-plan/v1-ras/spark32/11.txt | 422 +++
.../tpch-approved-plan/v1-ras/spark32/12.txt | 287 ++
.../tpch-approved-plan/v1-ras/spark32/13.txt | 304 ++
.../tpch-approved-plan/v1-ras/spark32/14.txt | 207 ++
.../tpch-approved-plan/v1-ras/spark32/15.txt | 266 ++
.../tpch-approved-plan/v1-ras/spark32/16.txt | 379 +++
.../tpch-approved-plan/v1-ras/spark32/17.txt | 343 ++
.../tpch-approved-plan/v1-ras/spark32/18.txt | 581 ++++
.../tpch-approved-plan/v1-ras/spark32/19.txt | 202 ++
.../tpch-approved-plan/v1-ras/spark32/20.txt | 735 ++++
.../tpch-approved-plan/v1-ras/spark32/21.txt | 708 ++++
.../tpch-approved-plan/v1-ras/spark32/22.txt | 270 ++
.../tpch-approved-plan/v1-ras/spark32/3.txt | 347 ++
.../tpch-approved-plan/v1-ras/spark32/4.txt | 292 ++
.../tpch-approved-plan/v1-ras/spark32/5.txt | 792 +++++
.../tpch-approved-plan/v1-ras/spark32/6.txt | 116 +
.../tpch-approved-plan/v1-ras/spark32/7.txt | 754 +++++
.../tpch-approved-plan/v1-ras/spark32/8.txt | 1047 ++++++
.../tpch-approved-plan/v1-ras/spark32/9.txt | 787 +++++
.../tpch-approved-plan/v1-ras/spark33/1.txt | 162 +
.../tpch-approved-plan/v1-ras/spark33/10.txt | 516 +++
.../tpch-approved-plan/v1-ras/spark33/11.txt | 701 ++++
.../tpch-approved-plan/v1-ras/spark33/12.txt | 287 ++
.../tpch-approved-plan/v1-ras/spark33/13.txt | 304 ++
.../tpch-approved-plan/v1-ras/spark33/14.txt | 207 ++
.../tpch-approved-plan/v1-ras/spark33/15.txt | 408 +++
.../tpch-approved-plan/v1-ras/spark33/16.txt | 379 +++
.../tpch-approved-plan/v1-ras/spark33/17.txt | 343 ++
.../tpch-approved-plan/v1-ras/spark33/18.txt | 581 ++++
.../tpch-approved-plan/v1-ras/spark33/19.txt | 202 ++
.../tpch-approved-plan/v1-ras/spark33/20.txt | 724 ++++
.../tpch-approved-plan/v1-ras/spark33/21.txt | 703 ++++
.../tpch-approved-plan/v1-ras/spark33/22.txt | 410 +++
.../tpch-approved-plan/v1-ras/spark33/3.txt | 347 ++
.../tpch-approved-plan/v1-ras/spark33/4.txt | 292 ++
.../tpch-approved-plan/v1-ras/spark33/5.txt | 792 +++++
.../tpch-approved-plan/v1-ras/spark33/6.txt | 116 +
.../tpch-approved-plan/v1-ras/spark33/7.txt | 754 +++++
.../tpch-approved-plan/v1-ras/spark33/8.txt | 1047 ++++++
.../tpch-approved-plan/v1-ras/spark33/9.txt | 787 +++++
.../tpch-approved-plan/v1-ras/spark34/1.txt | 162 +
.../tpch-approved-plan/v1-ras/spark34/10.txt | 522 +++
.../tpch-approved-plan/v1-ras/spark34/11.txt | 709 ++++
.../tpch-approved-plan/v1-ras/spark34/12.txt | 289 ++
.../tpch-approved-plan/v1-ras/spark34/13.txt | 306 ++
.../tpch-approved-plan/v1-ras/spark34/14.txt | 209 ++
.../tpch-approved-plan/v1-ras/spark34/15.txt | 410 +++
.../tpch-approved-plan/v1-ras/spark34/16.txt | 382 +++
.../tpch-approved-plan/v1-ras/spark34/17.txt | 347 ++
.../tpch-approved-plan/v1-ras/spark34/18.txt | 589 ++++
.../tpch-approved-plan/v1-ras/spark34/19.txt | 204 ++
.../tpch-approved-plan/v1-ras/spark34/20.txt | 734 ++++
.../tpch-approved-plan/v1-ras/spark34/21.txt | 713 ++++
.../tpch-approved-plan/v1-ras/spark34/22.txt | 412 +++
.../tpch-approved-plan/v1-ras/spark34/3.txt | 351 ++
.../tpch-approved-plan/v1-ras/spark34/4.txt | 294 ++
.../tpch-approved-plan/v1-ras/spark34/5.txt | 802 +++++
.../tpch-approved-plan/v1-ras/spark34/6.txt | 116 +
.../tpch-approved-plan/v1-ras/spark34/7.txt | 764 +++++
.../tpch-approved-plan/v1-ras/spark34/8.txt | 1061 ++++++
.../tpch-approved-plan/v1-ras/spark34/9.txt | 797 +++++
.../tpch-approved-plan/v1-ras/spark35/1.txt | 162 +
.../tpch-approved-plan/v1-ras/spark35/10.txt | 522 +++
.../tpch-approved-plan/v1-ras/spark35/11.txt | 709 ++++
.../tpch-approved-plan/v1-ras/spark35/12.txt | 289 ++
.../tpch-approved-plan/v1-ras/spark35/13.txt | 306 ++
.../tpch-approved-plan/v1-ras/spark35/14.txt | 209 ++
.../tpch-approved-plan/v1-ras/spark35/15.txt | 410 +++
.../tpch-approved-plan/v1-ras/spark35/16.txt | 382 +++
.../tpch-approved-plan/v1-ras/spark35/17.txt | 347 ++
.../tpch-approved-plan/v1-ras/spark35/18.txt | 589 ++++
.../tpch-approved-plan/v1-ras/spark35/19.txt | 204 ++
.../tpch-approved-plan/v1-ras/spark35/20.txt | 734 ++++
.../tpch-approved-plan/v1-ras/spark35/21.txt | 713 ++++
.../tpch-approved-plan/v1-ras/spark35/22.txt | 412 +++
.../tpch-approved-plan/v1-ras/spark35/3.txt | 351 ++
.../tpch-approved-plan/v1-ras/spark35/4.txt | 294 ++
.../tpch-approved-plan/v1-ras/spark35/5.txt | 802 +++++
.../tpch-approved-plan/v1-ras/spark35/6.txt | 116 +
.../tpch-approved-plan/v1-ras/spark35/7.txt | 764 +++++
.../tpch-approved-plan/v1-ras/spark35/8.txt | 1061 ++++++
.../tpch-approved-plan/v1-ras/spark35/9.txt | 797 +++++
.../resources/tpch-approved-plan/v1/spark32/1.txt | 162 +
.../resources/tpch-approved-plan/v1/spark32/10.txt | 516 +++
.../resources/tpch-approved-plan/v1/spark32/11.txt | 422 +++
.../resources/tpch-approved-plan/v1/spark32/12.txt | 287 ++
.../resources/tpch-approved-plan/v1/spark32/13.txt | 304 ++
.../resources/tpch-approved-plan/v1/spark32/14.txt | 207 ++
.../resources/tpch-approved-plan/v1/spark32/15.txt | 266 ++
.../resources/tpch-approved-plan/v1/spark32/16.txt | 379 +++
.../resources/tpch-approved-plan/v1/spark32/17.txt | 343 ++
.../resources/tpch-approved-plan/v1/spark32/18.txt | 581 ++++
.../resources/tpch-approved-plan/v1/spark32/19.txt | 202 ++
.../resources/tpch-approved-plan/v1/spark32/20.txt | 735 ++++
.../resources/tpch-approved-plan/v1/spark32/21.txt | 708 ++++
.../resources/tpch-approved-plan/v1/spark32/22.txt | 270 ++
.../resources/tpch-approved-plan/v1/spark32/3.txt | 347 ++
.../resources/tpch-approved-plan/v1/spark32/4.txt | 292 ++
.../resources/tpch-approved-plan/v1/spark32/5.txt | 792 +++++
.../resources/tpch-approved-plan/v1/spark32/6.txt | 116 +
.../resources/tpch-approved-plan/v1/spark32/7.txt | 754 +++++
.../resources/tpch-approved-plan/v1/spark32/8.txt | 1047 ++++++
.../resources/tpch-approved-plan/v1/spark32/9.txt | 787 +++++
.../resources/tpch-approved-plan/v1/spark33/1.txt | 162 +
.../resources/tpch-approved-plan/v1/spark33/10.txt | 516 +++
.../resources/tpch-approved-plan/v1/spark33/11.txt | 701 ++++
.../resources/tpch-approved-plan/v1/spark33/12.txt | 287 ++
.../resources/tpch-approved-plan/v1/spark33/13.txt | 304 ++
.../resources/tpch-approved-plan/v1/spark33/14.txt | 207 ++
.../resources/tpch-approved-plan/v1/spark33/15.txt | 408 +++
.../resources/tpch-approved-plan/v1/spark33/16.txt | 379 +++
.../resources/tpch-approved-plan/v1/spark33/17.txt | 343 ++
.../resources/tpch-approved-plan/v1/spark33/18.txt | 581 ++++
.../resources/tpch-approved-plan/v1/spark33/19.txt | 202 ++
.../resources/tpch-approved-plan/v1/spark33/20.txt | 724 ++++
.../resources/tpch-approved-plan/v1/spark33/21.txt | 703 ++++
.../resources/tpch-approved-plan/v1/spark33/22.txt | 410 +++
.../resources/tpch-approved-plan/v1/spark33/3.txt | 347 ++
.../resources/tpch-approved-plan/v1/spark33/4.txt | 292 ++
.../resources/tpch-approved-plan/v1/spark33/5.txt | 792 +++++
.../resources/tpch-approved-plan/v1/spark33/6.txt | 116 +
.../resources/tpch-approved-plan/v1/spark33/7.txt | 754 +++++
.../resources/tpch-approved-plan/v1/spark33/8.txt | 1047 ++++++
.../resources/tpch-approved-plan/v1/spark33/9.txt | 787 +++++
.../resources/tpch-approved-plan/v1/spark34/1.txt | 162 +
.../resources/tpch-approved-plan/v1/spark34/10.txt | 522 +++
.../resources/tpch-approved-plan/v1/spark34/11.txt | 709 ++++
.../resources/tpch-approved-plan/v1/spark34/12.txt | 289 ++
.../resources/tpch-approved-plan/v1/spark34/13.txt | 306 ++
.../resources/tpch-approved-plan/v1/spark34/14.txt | 209 ++
.../resources/tpch-approved-plan/v1/spark34/15.txt | 410 +++
.../resources/tpch-approved-plan/v1/spark34/16.txt | 382 +++
.../resources/tpch-approved-plan/v1/spark34/17.txt | 347 ++
.../resources/tpch-approved-plan/v1/spark34/18.txt | 589 ++++
.../resources/tpch-approved-plan/v1/spark34/19.txt | 204 ++
.../resources/tpch-approved-plan/v1/spark34/20.txt | 734 ++++
.../resources/tpch-approved-plan/v1/spark34/21.txt | 713 ++++
.../resources/tpch-approved-plan/v1/spark34/22.txt | 412 +++
.../resources/tpch-approved-plan/v1/spark34/3.txt | 351 ++
.../resources/tpch-approved-plan/v1/spark34/4.txt | 294 ++
.../resources/tpch-approved-plan/v1/spark34/5.txt | 802 +++++
.../resources/tpch-approved-plan/v1/spark34/6.txt | 116 +
.../resources/tpch-approved-plan/v1/spark34/7.txt | 764 +++++
.../resources/tpch-approved-plan/v1/spark34/8.txt | 1061 ++++++
.../resources/tpch-approved-plan/v1/spark34/9.txt | 797 +++++
.../resources/tpch-approved-plan/v1/spark35/1.txt | 162 +
.../resources/tpch-approved-plan/v1/spark35/10.txt | 522 +++
.../resources/tpch-approved-plan/v1/spark35/11.txt | 709 ++++
.../resources/tpch-approved-plan/v1/spark35/12.txt | 289 ++
.../resources/tpch-approved-plan/v1/spark35/13.txt | 306 ++
.../resources/tpch-approved-plan/v1/spark35/14.txt | 209 ++
.../resources/tpch-approved-plan/v1/spark35/15.txt | 410 +++
.../resources/tpch-approved-plan/v1/spark35/16.txt | 382 +++
.../resources/tpch-approved-plan/v1/spark35/17.txt | 347 ++
.../resources/tpch-approved-plan/v1/spark35/18.txt | 589 ++++
.../resources/tpch-approved-plan/v1/spark35/19.txt | 204 ++
.../resources/tpch-approved-plan/v1/spark35/20.txt | 734 ++++
.../resources/tpch-approved-plan/v1/spark35/21.txt | 713 ++++
.../resources/tpch-approved-plan/v1/spark35/22.txt | 412 +++
.../resources/tpch-approved-plan/v1/spark35/3.txt | 351 ++
.../resources/tpch-approved-plan/v1/spark35/4.txt | 294 ++
.../resources/tpch-approved-plan/v1/spark35/5.txt | 802 +++++
.../resources/tpch-approved-plan/v1/spark35/6.txt | 116 +
.../resources/tpch-approved-plan/v1/spark35/7.txt | 764 +++++
.../resources/tpch-approved-plan/v1/spark35/8.txt | 1061 ++++++
.../resources/tpch-approved-plan/v1/spark35/9.txt | 797 +++++
.../gluten/config/AllBoltConfiguration.scala | 84 +
.../execution/BoltAggregateFunctionsSuite.scala | 1377 ++++++++
.../gluten/execution/BoltColumnarCacheSuite.scala | 247 ++
.../execution/BoltExplodeExpressionSuite.scala | 178 +
.../gluten/execution/BoltHashJoinSuite.scala | 310 ++
.../apache/gluten/execution/BoltLiteralSuite.scala | 142 +
.../apache/gluten/execution/BoltMetricsSuite.scala | 310 ++
.../execution/BoltOrcDataTypeValidationSuite.scala | 483 +++
.../BoltParquetDataTypeValidationSuite.scala | 490 +++
.../apache/gluten/execution/BoltScanSuite.scala | 307 ++
.../execution/BoltStringFunctionsSuite.scala | 690 ++++
.../apache/gluten/execution/BoltTPCDSSuite.scala | 132 +
.../execution/BoltWholeStageTransformerSuite.scala | 11 +-
.../execution/BoltWindowExpressionSuite.scala | 174 +
.../apache/gluten/execution/FallbackSuite.scala | 370 ++
.../gluten/execution/MiscOperatorSuite.scala | 2357 +++++++++++++
.../python/ArrowEvalPythonExecSuite.scala | 126 +
.../apache/gluten/expression/BoltUdfSuite.scala | 286 ++
.../gluten/expression/UDFPartialProjectSuite.scala | 276 ++
.../columnar/transition/BoltTransitionSuite.scala | 242 ++
.../functions/DateFunctionsValidateSuite.scala | 492 +++
.../functions/ScalarFunctionsValidateSuite.scala | 1545 +++++++++
.../gluten/utils/MySharedLibraryLoader.scala | 30 +-
.../utils/ParquetEncryptionDetectionSuite.scala | 233 ++
.../utils/SharedLibraryLoaderUtilsSuite.scala | 25 +-
.../sql/catalyst/expressions/BoltCastSuite.scala | 84 +
.../spark/sql/execution/BoltExpandSuite.scala | 80 +
.../spark/sql/execution/BoltLocalCacheSuite.scala | 56 +
.../spark/sql/execution/BoltParquetReadSuite.scala | 54 +
.../execution/BoltParquetWriteForHiveSuite.scala | 448 +++
.../sql/execution/BoltParquetWriteSuite.scala | 167 +
.../spark/sql/execution/BucketWriteUtils.scala | 95 +
.../spark/sql/execution/GlutenHiveUDFSuite.scala | 393 +++
.../benchmark/ColumnarTableCacheBenchmark.scala | 88 +
.../execution/joins/GlutenExistenceJoinSuite.scala | 100 +
.../UnsafeColumnarBuildSideRelationTest.scala | 96 +
.../AutoAdjustStageResourceProfileSuite.scala | 4 +-
.../execution/DynamicOffHeapSizingSuite.scala | 2 +-
.../spark/memory/GlobalOffHeapMemorySuite.scala | 0
build.md | 209 ++
cpp/CMakeLists.txt | 7 +-
cpp/CMakeUserPresets.json | 9 +
cpp/{CMakeLists.txt => bolt.CMakeLists.cmake} | 71 +-
cpp/bolt/CMakeLists.txt | 413 +++
cpp/bolt/benchmarks/CMakeLists.txt | 41 +
cpp/bolt/benchmarks/ExtractNullBitsBenchmark.cc | 86 +
cpp/bolt/benchmarks/GenericBenchmark.cc | 766 +++++
cpp/bolt/benchmarks/ParquetWriteBenchmark.cc | 156 +
cpp/bolt/benchmarks/PlanValidatorUtil.cc | 66 +
cpp/bolt/benchmarks/common/BenchmarkUtils.cc | 166 +
cpp/bolt/benchmarks/common/BenchmarkUtils.h | 97 +
cpp/bolt/benchmarks/data/plan/q17_joins.json | 601 ++++
cpp/bolt/benchmarks/exec/OrcConverter.cc | 107 +
.../bolt/bolt-build-info.sh | 19 +-
cpp/bolt/compute/BoltBackend.cc | 495 +++
cpp/bolt/compute/BoltBackend.h | 130 +
cpp/bolt/compute/BoltPlanConverter.cc | 161 +
cpp/bolt/compute/BoltPlanConverter.h | 55 +
cpp/bolt/compute/BoltRuntime.cc | 382 +++
cpp/bolt/compute/BoltRuntime.h | 151 +
cpp/bolt/compute/TaskStatusListener.cc | 172 +
cpp/bolt/compute/TaskStatusListener.h | 63 +
cpp/bolt/compute/WholeStageResultIterator.cc | 1039 ++++++
cpp/bolt/compute/WholeStageResultIterator.h | 173 +
.../compute/iceberg/IcebergFormat.cc} | 24 +-
.../compute/iceberg/IcebergFormat.h} | 14 +-
cpp/bolt/compute/iceberg/IcebergWriter.cc | 193 ++
cpp/bolt/compute/iceberg/IcebergWriter.h | 59 +
cpp/bolt/compute/paimon/PaimonPlanUtils.cc | 66 +
cpp/bolt/compute/paimon/PaimonPlanUtils.h | 76 +
cpp/bolt/config/BoltConfig.h | 268 ++
cpp/bolt/cudf/CudfPlanValidator.cc | 79 +
.../Likely.h => bolt/cudf/CudfPlanValidator.h} | 8 +-
cpp/bolt/jni/BoltJniWrapper.cc | 1173 +++++++
cpp/bolt/jni/JniFileSystem.cc | 477 +++
.../utils/Likely.h => bolt/jni/JniFileSystem.h} | 21 +-
cpp/{core/utils/Likely.h => bolt/jni/JniUdf.cc} | 20 +-
cpp/{core/utils/Likely.h => bolt/jni/JniUdf.h} | 18 +-
cpp/bolt/memory/ArrowMemory.h | 53 +
cpp/bolt/memory/BoltColumnarBatch.cc | 173 +
cpp/bolt/memory/BoltColumnarBatch.h | 68 +
cpp/bolt/memory/BoltMemoryManager.cc | 474 +++
cpp/bolt/memory/BoltMemoryManager.h | 125 +
.../bolt/memory/BufferOutputStream.cc | 34 +-
.../Likely.h => bolt/memory/BufferOutputStream.h} | 30 +-
.../operators/functions/Arithmetic.h} | 11 +-
.../functions/RegistrationAllFunctions.cc | 104 +
.../functions/RegistrationAllFunctions.h} | 8 +-
.../operators/functions/RowConstructorWithNull.cc | 57 +
.../operators/functions/RowConstructorWithNull.h | 54 +
cpp/bolt/operators/functions/RowFunctionWithNull.h | 85 +
.../functions/SparkExprToSubfieldFilterParser.cc | 100 +
.../functions/SparkExprToSubfieldFilterParser.h} | 22 +-
cpp/bolt/operators/plannodes/RowVectorStream.h | 240 ++
cpp/bolt/operators/reader/FileReaderIterator.cc | 47 +
.../operators/reader/FileReaderIterator.h} | 37 +-
cpp/bolt/operators/reader/ParquetReaderIterator.cc | 136 +
cpp/bolt/operators/reader/ParquetReaderIterator.h | 77 +
.../serializer/BoltColumnarBatchSerializer.cc | 117 +
.../serializer/BoltColumnarBatchSerializer.h | 57 +
.../serializer/BoltColumnarToRowConverter.cc | 91 +
.../serializer/BoltColumnarToRowConverter.h | 48 +
.../serializer/BoltRowToColumnarConverter.cc | 310 ++
.../serializer/BoltRowToColumnarConverter.h} | 30 +-
.../operators/writer/BoltColumnarBatchWriter.cc | 59 +
.../operators/writer/BoltColumnarBatchWriter.h} | 35 +-
.../operators/writer/BoltDataSource.h} | 14 +-
cpp/bolt/operators/writer/BoltParquetDataSource.cc | 245 ++
cpp/bolt/operators/writer/BoltParquetDataSource.h | 124 +
.../operators/writer/BoltParquetDataSourceABFS.h | 56 +
.../operators/writer/BoltParquetDataSourceGCS.h | 53 +
.../operators/writer/BoltParquetDataSourceHDFS.h | 53 +
.../operators/writer/BoltParquetDataSourceS3.h | 53 +
cpp/bolt/shuffle/BoltShuffleReaderWrapper.h | 136 +
cpp/bolt/shuffle/BoltShuffleWriterWrapper.h | 241 ++
cpp/bolt/shuffle/ReaderStreamIteratorWrapper.h | 68 +
cpp/bolt/shuffle/RssClientWrapper.h | 47 +
.../Likely.h => bolt/shuffle/SparkInputStream.h} | 24 +-
cpp/bolt/substrait/BoltSubstraitSignature.cc | 265 ++
cpp/bolt/substrait/BoltSubstraitSignature.h | 45 +
cpp/bolt/substrait/BoltToSubstraitExpr.cc | 623 ++++
cpp/bolt/substrait/BoltToSubstraitExpr.h | 108 +
cpp/bolt/substrait/BoltToSubstraitPlan.cc | 406 +++
cpp/bolt/substrait/BoltToSubstraitPlan.h | 114 +
cpp/bolt/substrait/BoltToSubstraitType.cc | 165 +
.../substrait/BoltToSubstraitType.h} | 30 +-
cpp/bolt/substrait/SubstraitExtensionCollector.cc | 41 +
cpp/bolt/substrait/SubstraitExtensionCollector.h | 50 +
cpp/bolt/substrait/SubstraitParser.cc | 470 +++
cpp/bolt/substrait/SubstraitParser.h | 123 +
cpp/bolt/substrait/SubstraitToBoltExpr.cc | 651 ++++
cpp/bolt/substrait/SubstraitToBoltExpr.h | 114 +
cpp/bolt/substrait/SubstraitToBoltPlan.cc | 1605 +++++++++
cpp/bolt/substrait/SubstraitToBoltPlan.h | 313 ++
cpp/bolt/substrait/SubstraitToBoltPlanValidator.cc | 1445 ++++++++
cpp/bolt/substrait/SubstraitToBoltPlanValidator.h | 175 +
cpp/bolt/substrait/TypeUtils.h | 96 +
.../substrait/VariantToVectorConverter.cc} | 18 +-
.../substrait/VariantToVectorConverter.h} | 12 +-
cpp/bolt/symbols.map | 16 +
cpp/bolt/tests/BoltBatchResizerTest.cc | 86 +
cpp/bolt/tests/BoltColumnarBatchTest.cc | 65 +
cpp/bolt/tests/BoltColumnarToRowTest.cc | 97 +
cpp/bolt/tests/BoltRowToColumnarTest.cc | 146 +
cpp/bolt/tests/BoltSubstraitRoundTripTest.cc | 556 +++
cpp/bolt/tests/BoltSubstraitSignatureTest.cc | 167 +
cpp/bolt/tests/BoltToSubstraitTypeTest.cc | 65 +
cpp/bolt/tests/BufferOutputStreamTest.cc | 75 +
cpp/bolt/tests/CMakeLists.txt | 83 +
.../Likely.h => bolt/tests/FilePathGenerator.cc} | 16 +-
.../Likely.h => bolt/tests/FilePathGenerator.h} | 8 +-
cpp/bolt/tests/FunctionTest.cc | 222 ++
.../tests/JsonToProtoConverter.cc} | 10 +-
.../Likely.h => bolt/tests/JsonToProtoConverter.h} | 8 +-
cpp/bolt/tests/MemoryManagerTest.cc | 414 +++
cpp/bolt/tests/MyUdfTest.cc | 42 +
cpp/bolt/tests/OrcTest.cc | 172 +
cpp/bolt/tests/RuntimeTest.cc | 163 +
cpp/bolt/tests/SparkFunctionTest.cc | 113 +
cpp/bolt/tests/Substrait2BoltPlanConversionTest.cc | 289 ++
cpp/bolt/tests/Substrait2BoltPlanValidatorTest.cc | 65 +
.../Substrait2BoltValuesNodeConversionTest.cc | 62 +
cpp/bolt/tests/SubstraitExtensionCollectorTest.cc | 123 +
cpp/bolt/tests/data/filter_upper.json | 126 +
cpp/bolt/tests/data/filter_upper_split.json | 8 +
cpp/bolt/tests/data/group.json | 34 +
cpp/bolt/tests/data/if_then.json | 387 +++
cpp/bolt/tests/data/if_then_split.json | 8 +
cpp/bolt/tests/data/q1_first_stage.json | 877 +++++
cpp/bolt/tests/data/q6_first_stage.json | 592 ++++
cpp/bolt/tests/data/q6_first_stage_split.json | 11 +
cpp/bolt/tests/data/substrait_virtualTable.json | 153 +
cpp/bolt/tests/iceberg/IcebergWriteTest.cc | 71 +
cpp/bolt/tests/utils/TestAllocationListener.cc | 72 +
.../tests/utils/TestAllocationListener.h} | 8 +-
.../tests/utils/TestStreamReader.h} | 8 +-
.../Likely.h => bolt/tests/utils/TestUtils.h} | 8 +-
cpp/bolt/udf/BoltUdf.cc | 63 +
cpp/bolt/udf/BoltUdf.h | 101 +
cpp/{core/utils/Likely.h => bolt/udf/Udaf.h} | 8 +-
cpp/{core/utils/Likely.h => bolt/udf/Udf.h} | 8 +-
cpp/bolt/udf/UdfLoader.cc | 184 +
cpp/bolt/udf/UdfLoader.h | 98 +
.../bolt/udf/examples/CMakeLists.txt | 18 +-
cpp/bolt/udf/examples/MyUDAF.cc | 220 ++
cpp/bolt/udf/examples/MyUDF.cc | 115 +
cpp/bolt/udf/examples/UdfCommon.h | 53 +
cpp/bolt/utils/BoltArrowUtils.cc | 91 +
cpp/bolt/utils/BoltArrowUtils.h | 64 +
cpp/bolt/utils/BoltBatchResizer.cc | 127 +
cpp/bolt/utils/BoltBatchResizer.h | 47 +
cpp/bolt/utils/BoltWholeStageDumper.cc | 131 +
cpp/bolt/utils/BoltWholeStageDumper.h | 51 +
cpp/bolt/utils/BoltWriterUtils.cc | 119 +
.../Likely.h => bolt/utils/BoltWriterUtils.h} | 17 +-
cpp/bolt/utils/Common.cc | 64 +
cpp/bolt/utils/Common.h | 51 +
cpp/bolt/utils/ConfigExtractor.cc | 315 ++
cpp/bolt/utils/ConfigExtractor.h | 50 +
.../Likely.h => bolt/utils/JsonToProtoConverter.h} | 8 +-
cpp/bolt/version/CMakeLists.txt | 82 +
cpp/bolt/version/version.h.in | 92 +
cpp/conanfile.py | 189 ++
cpp/core/CMakeLists.txt | 7 +-
cpp/core/benchmarks/CMakeLists.txt | 37 +
cpp/core/{CMakeLists.txt => bolt.CMakeLists.cmake} | 173 +-
cpp/core/compute/Runtime.h | 3 +
cpp/core/config/GlutenConfig.h | 3 +
cpp/core/jni/JniCommon.cc | 48 +-
cpp/core/jni/JniCommon.h | 345 +-
cpp/core/jni/JniWrapper.cc | 50 +-
cpp/core/{utils/Likely.h => jni/JniWrapper.h} | 18 +-
cpp/core/memory/BoltGlutenMemoryManager.cc | 303 ++
cpp/core/memory/BoltGlutenMemoryManager.h | 99 +
cpp/core/memory/OnHeapUsageGetter.h | 131 +
cpp/core/nativeLoader/CMakeLists.txt | 46 +
cpp/core/nativeLoader/JniNativeLibraryLoader.cc | 153 +
cpp/core/shuffle/ShuffleReaderBase.h | 65 +
cpp/core/shuffle/ShuffleWriterBase.h | 82 +
cpp/core/utils/ConfigResolver.cc | 42 +
cpp/core/utils/{Likely.h => ConfigResolver.h} | 25 +-
cpp/core/utils/Likely.h | 8 +-
cpp/core/utils/TestUtils.h | 43 +
dev/build_bolt_arrow.sh | 83 +
dev/docker/Dockerfile.centos8-bolt | 123 +
dev/docker/Dockerfile.ubuntu22-bolt | 65 +
dev/gen-all-config-docs.sh | 6 +
dev/gluten-build-info.sh | 30 +
dev/install-conan.sh | 69 +
dev/install-gcc.sh | 54 +
docs/bolt-backend-aggregate-function-support.md | 70 +
docs/bolt-backend-generator-function-support.md | 16 +
docs/bolt-backend-scalar-function-support.md | 391 +++
docs/bolt-backend-window-function-support.md | 13 +
docs/bolt-configuration.md | 87 +
docs/bolt-parquet-write-configuration.md | 76 +
docs/bolt-quick-start.md | 121 +
docs/bolt-spark-configuration.md | 3541 ++++++++++++++++++++
docs/developers/bolt-function-development-guide.md | 242 ++
docs/get-started/Bolt.md | 6 +
docs/get-started/BoltFileSystem.md | 128 +
docs/get-started/BoltLocalCache.md | 79 +
docs/get-started/BoltStageResourceAdj.md | 75 +
docs/image/bolt_stage_resource.png | Bin 0 -> 124896 bytes
docs/image/boltbe_sqlframe.png | Bin 0 -> 143419 bytes
docs/velox-to-bolt-migration-guide.md | 160 +
.../gluten/execution/PaimonSparkShimImpl.scala | 35 +
.../gluten/substrait/rel/PaimonLocalFilesNode.java | 21 +-
.../gluten/execution/PaimonScanTransformer.scala | 7 +-
.../apache/gluten/execution/PaimonSparkShim.scala | 25 +
.../substrait/proto/substrait/algebra.proto | 5 +-
.../apache/gluten/backendsapi/IteratorApi.scala | 10 +
.../org/apache/gluten/config/GlutenConfig.scala | 221 +-
.../apache/gluten/utils/BackendTestSettings.scala | 6 +
.../org/apache/gluten/utils/BackendTestUtils.scala | 4 +
gluten-ut/spark33/pom.xml | 15 +
.../utils/bolt/BoltSQLQueryTestSettings.scala | 265 ++
.../gluten/utils/bolt/BoltTestSettings.scala | 862 +++++
.../apache/spark/sql/GlutenCachedTableSuite.scala | 7 +-
.../spark/sql/GlutenDataFrameFunctionsSuite.scala | 33 +
.../sql/GlutenDynamicPartitionPruningSuite.scala | 4 +-
.../GlutenCollectionExpressionsSuite.scala | 2 +-
.../adaptive/bolt/BoltAdaptiveQueryExecSuite.scala | 1600 +++++++++
.../hive/execution/GlutenHiveSQLQuerySuite.scala | 2 +-
gluten-ut/spark34/pom.xml | 38 +
.../utils/bolt/BoltSQLQueryTestSettings.scala | 268 ++
.../gluten/utils/bolt/BoltTestSettings.scala | 904 +++++
.../apache/spark/sql/GlutenCachedTableSuite.scala | 7 +-
.../spark/sql/GlutenDataFrameFunctionsSuite.scala | 33 +
.../GlutenCollectionExpressionsSuite.scala | 2 +-
.../adaptive/bolt/BoltAdaptiveQueryExecSuite.scala | 1549 +++++++++
.../spark/sql/gluten/GlutenFallbackSuite.scala | 4 +-
gluten-ut/spark35/pom.xml | 60 +
.../gluten/GlutenColumnarWriteTestSupport.scala | 17 +-
.../utils/bolt/BoltSQLQueryTestSettings.scala | 296 ++
.../gluten/utils/bolt/BoltTestSettings.scala | 987 ++++++
.../apache/spark/sql/GlutenCachedTableSuite.scala | 7 +-
.../spark/sql/GlutenDataFrameFunctionsSuite.scala | 33 +
.../apache/spark/sql/GlutenDataFrameSuite.scala | 4 +-
.../GlutenCollectionExpressionsSuite.scala | 2 +-
.../adaptive/bolt/BoltAdaptiveQueryExecSuite.scala | 1549 +++++++++
.../spark/sql/gluten/GlutenFallbackSuite.scala | 4 +-
.../spark/sql/GlutenDateFunctionsSuite.scala | 28 +-
.../GlutenCollectionExpressionsSuite.scala | 2 +-
gluten-ut/test/pom.xml | 15 +
package/pom.xml | 62 +-
plan.md | 935 ++++++
pom.xml | 13 +
.../scala/org/apache/gluten/GlutenBuildInfo.scala | 3 +
853 files changed, 235327 insertions(+), 1105 deletions(-)
diff --git a/.github/skills/extract-cpp-common/SKILL.md b/.github/skills/extract-cpp-common/SKILL.md
new file mode 100644
index 0000000000..9bc4feda35
--- /dev/null
+++ b/.github/skills/extract-cpp-common/SKILL.md
@@ -0,0 +1,87 @@
+---
+name: extract-cpp-common
+description: >-
+ 当用户需要把 cpp/ 下多个 backend（如 bolt / velox）中“逻辑完全一致、仅 backend 符号名不同”的原生代码抽取到公共目录 cpp/core，以消除未来 rebase 冲突时，使用本技能。强调“仅忽略符号命名差异、零行为变更、公共落 cpp/core、backend 留壳”。触发词：`/extract-cpp-common`。
+argument-hint: "[ <file/dir/module ...> ] 例: cpp/bolt/substrait/VariantToVectorConverter.cc 或 cpp/bolt/jni/ 或 jni"
+user-invocable: true
+---
+
+# 抽取 cpp/ 公共代码 Skill
+
+把 `cpp/` 下多个 backend 里**逐字一致、仅 backend 符号名不同**的代码，抽到公共目录 `cpp/core`，
+让未来各 backend rebase 不再在这些文件上冲突。**核心准则：零行为变更，能逐字搬运就不重写。**
+
+## 适用范围（必须支持）
+
+用户调用时可以指定代码范围，本 Skill 必须遵守：
+- **某几个文件**：只评估并抽取这些文件。
+- **某些目录**：扫描目录下候选并逐个判定。
+- **某个模块**：先把模块映射到 `cpp/<backend>/<...>` 的文件集合，再判定。
+- **留空**：要求用户给范围或先给出候选清单确认，不要全仓盲扫硬抽。
+
+开始前先复述理解的范围与涉及的 backend（如 bolt + velox）。
+
+## 第 0 步 — 抽取口径（最重要，先卡死）
+
+唯一允许忽略的差异：**backend 符号命名**，例如
+`Bolt`/`Velox`、`bytedance::bolt`/`facebook::velox`、`BOLT_*`/`VELOX_*` 宏、backend 头路径。
+**其余逻辑必须逐字一致才抽**。分档：
+- 第一档：100% 一致仅符号名不同 → 低风险，本 Skill 主战场。
+- “像但已分叉”（主 plan converter / memory manager / shuffle runtime / 主转换链）→ 放 backlog，**不抽**。
+
+## 第 1 步 — 致命前置检查：依赖是否真实存在（避免返工）
+
+抽之前，对每个候选、每个 backend 都确认它真的能编：
+1. `grep` 候选里依赖的关键符号（类型 / 命名空间 / 宏）在该 backend 的头或 conan 包里是否存在。
+2. 看该 backend 的 `CMakeLists.txt`：该源文件是否被注释 / 不参与编译。
+3. 看上层引用（如 `*PlanConverter` / `WholeStageResultIterator`）对它的 include 是否也是注释。
+
+反例教训：`IcebergPlanConverter` 在 bolt 侧依赖的 iceberg connector 在 bolt conan 包里根本不存在，
+文件本就被注释、从不编译——这类**某 backend 缺依赖**的文件不要纳入该 backend 的抽取，
+应只在有依赖的 backend + core 抽，或直接删除该 backend 的死文件（先确认引用全是注释再删）。
+
+## 第 2 步 — 逐字对比确认一致
+
+对每对 backend 文件做 diff，把已知的 backend 符号差异归一后，确认剩余**逐字一致**。
+有任何实质逻辑差异 → 退回 backlog，不抽。
+
+## 第 3 步 — 选择抽取手法（按依赖深度，从轻到重）
+
+1. **纯声明 / 无 backend 依赖** → 直接下沉到 `cpp/core/...` 头，backend 头 `#include` 它。最干净。
+ （例：`registerJolFileSystem(uint64_t)` 这种无类型依赖声明，core 头 `#include <cstdint>` 即自洽。）
+2. **实现依赖 backend 类型（靠 `using namespace` 才可见）** → 公共实现放 `cpp/core`，
+ backend 文件先 `using namespace bytedance::bolt;`（或 velox）再 `#include` core 头/实现。
+3. **实现还依赖 backend 宏** → 用 `GLUTEN_*` 占位宏参数化：
+ - core `.cc` 顶部 `#ifndef GLUTEN_x ... #error` 强制约定；用 `GLUTEN_x` 写逻辑。
+ - backend `.cc`：先 include backend 头 → `#define GLUTEN_x BOLT_x` → `#include "../../core/.../X.cc"` → `#undef`。
+ - 说明：`#include .cc` 是反模式，仅在“逐字搬运比重写更安全”时采用，并要能讲清替代方案
+ （core 头里重写一份不依赖 backend 宏的实现），由用户取舍。
+
+统一规则：
+- 公共文件**一律落 `cpp/core`**（不要新建别的公共目录）。
+- backend 侧**保留原接口名与调用链**（如 `initBoltJniFileSystem` / `using VeloxDataSource = GlutenDataSource`），上层零感知。
+- 若公共 `.cc` 是被 backend `.cc` 文本复用的，则**不需要**改 `cpp/core/CMakeLists.txt`（编译入口仍由各 backend target 持有）；
+ 只有当 core 真正独立编译某 `.cc` 时才往 core target 加 source。
+
+## 第 4 步 — 工作流纪律（老规矩）
+
+- **公共抽取必走独立 worktree**：基于干净基线分支（如 `gluten.dev` 的 `fake_main`）新建 worktree + 专属分支，
+ 在那里成型 patch；主仓只做对齐。先确认基线分支与 worktree 命名后再建。
+- **认清基线差异**：社区基线可能没有私有 backend（如 bolt 私有）。社区 patch 通常只含 `core + <公开 backend>`；
+ 私有 backend 的薄包装在主仓单独手做，**不要混进公共 patch**。
+- **范围克制**：用户说只动 cpp 就不碰其他；回退要干净（`git rm` 死文件 + `git checkout` 还原 + 删空目录）。
+
+## 第 5 步 — 验证闭环
+
+- **先窄后宽**：用 `compile_commands.json` 抠出受影响文件的真实编译命令做增量编译定位问题，
+ 再按需 `make release` / 全量构建收口。
+- 改了头文件要 `touch` 强制重编，避免 “no work to do” 假通过。
+- 提交前 `git diff --check` 查空白；commit message 讲清“为什么”（消除 rebase 冲突 / 零行为变更）并带 `Generated-by`。
+
+## 第 6 步 — 状态留痕
+
+把“抽了哪些 / 暂缓哪些 / 为何暂缓 / 缺依赖证据路径”记入 `plan.md` 对应章节，便于跨 session 续作与回溯。
+
+## 一句话提炼
+
+**先验证依赖再动手 → 逐字确认仅符号差 → 按依赖深度选最轻手法（声明<类型<宏注入）→ 公共落 cpp/core、backend 留壳 → 独立 worktree 出 patch → 增量编译打底全量收口 → 决策入档。** 风险判断永远先于代码技巧。
diff --git a/.github/skills/extract-jvm-common/SKILL.md b/.github/skills/extract-jvm-common/SKILL.md
new file mode 100644
index 0000000000..33892fe01a
--- /dev/null
+++ b/.github/skills/extract-jvm-common/SKILL.md
@@ -0,0 +1,118 @@
+---
+name: extract-jvm-common
+description: >-
+ 当用户需要把多个 backend（如 bolt / velox / clickhouse）中“逻辑完全一致或高度相似、仅 backend 符号名/常量不同”的 Java/Scala 代码抽取到公共模块 backends-core（或 gluten-substrait/gluten-core 等共享层），以消除重复与未来 rebase 冲突时，使用本技能。强调“以某基线 backend 为参照、仅忽略命名差异、零行为变更、公共层放抽象基类/通用实现、backend 留 thin 门面、一组一提交”。可针对指定文件、目录或模块。触发词：`/extract-jvm-common`。
+argument-hint: "[ <file/dir/module ...> ] 例: backends-bolt/src-celeborn/.../CelebornColumnarShuffleWriterFactory.scala 或 backends-bolt/src-iceberg/ 或 metrics"
+user-invocable: true
+---
+
+# 抽取 JVM（Java/Scala）公共代码 Skill
+
+把多个 backend（bolt / velox / clickhouse 等）里**逻辑一致或高度相似、仅 backend 符号名/常量不同**的
+Java/Scala 代码，抽到公共层（优先 `backends-core`，其次 `gluten-substrait` / `gluten-core` / `gluten-arrow` 等已有共享模块），
+backend 仅保留 thin 门面（常量、配置、本地特定逻辑）。**核心准则：零行为变更，以某基线 backend 为参照，能继承复用就不重写。**
+
+**硬性原则：所有“公共抽取”任务，必须从 `/data00/home/liyang.127/oap/gluten.dev` 的 `fake_main` 分支新建独立 git worktree + 专属分支来完成；禁止直接在当前主仓 worktree、已有脏工作区、其它对象库 worktree，或任何非 `/data00/home/liyang.127/oap/gluten.dev` `fake_main` 派生的工作区上做公共抽取。**
+
+## 适用范围（必须支持）
+
+用户调用时可指定范围，本 Skill 必须遵守：
+- **某几个文件**：只评估并抽取这些文件。
+- **某些目录**：扫描目录下候选并逐个判定。
+- **某个模块**（如 celeborn / iceberg / metrics / shuffle）：先把模块映射到各 backend 下的文件集合，再判定。
+- **留空**：要求用户给范围或先给出候选清单确认，不要全仓盲扫硬抽。
+
+开始前先复述理解的范围、参照基线 backend（如以 velox 为基线）、涉及的其他 backend、以及目标公共模块。
+
+## 第 0 步 — 抽取口径（最重要，先卡死）
+
+1. **选定基线 backend**：以最成熟/最完整的 backend（通常 velox）为参照，其他 backend 向它对齐。
+2. **唯一允许忽略的差异**：backend 符号命名 —— `Bolt`/`Velox`/`CH`、包名前缀、`BoltConfig`/`VeloxConfig`、
+ 常量字符串键名、本地实现类名。**其余逻辑必须一致或可参数化才抽**。分档：
+ - 第一档：规范化命名后 100% 一致 → 低风险，优先抽（如 Celeborn 工厂、TopNTransformer、序列化器抽象）。
+ - 第二档：主体一致、仅少量后端特定常量/钩子不同 → 抽**抽象基类**到公共层，backend 子类只填差异部分。
+ - 第三档：已实质分叉（核心转换链、plan 重写、内存管理）→ 放 backlog，**不抽**。
+3. **先做“净代码行数收益”检查**：预估抽取后“新增共享代码 + backend 壳代码 + 配套构建改动”的总行数，是否**严格小于**抽取前各 backend 重复实现的总行数。
+ - 若**不能减少整体代码行数**，或只是把重复代码平移成“共享基类 + 同样长的 thin 壳”而总行数不降，**则不抽取**。
+ - 只有在减少重复、减少维护面、且代码总量净下降时，才进入后续抽取流程。
+4. **依赖方向铁律**：`gluten-core/gluten-substrait → backends-core → 具体 backend`。
+ 公共层**绝不能**反向依赖任何具体 backend。抽取前确认被抽代码不引用 backend 私有符号；
+ 若引用，则把该符号也下沉或参数化（抽象方法 / 构造参数）。
+
+## 第 1 步 — 规范化对比确认一致（考古先于动手）
+
+对每组待抽文件，在各 backend 间做“去名称化” diff：
+```
+diff <(sed 's/Velox//g;s/Bolt//g;s/CH//g' A.scala) <(sed 's/Velox//g;s/Bolt//g;s/CH//g' B.scala)
+```
+- diff 为空 → 纯改名拷贝，第一档，直接抽。
+- 仅少量差异且都是“后端特定常量/类名/钩子” → 第二档，抽象基类 + 子类填空。
+- 有实质逻辑差异 → 退回 backlog，不抽，并记录原因。
+
+**务必同时考古 old commit**：用 `git show <add-backend-commit> -- <file>` 确认该 backend 文件原始形态，
+判断差异是“有意的后端特性”还是“当初照搬后上游演进造成的脱节”。后者优先按基线 backend 对齐而非保留。
+
+## 第 2 步 — 致命前置检查：依赖与接口是否真实存在（避免返工）
+
+抽之前，对每个候选确认它在公共层与各 backend 都真能编：
+1. `grep` 候选依赖的关键符号（类 / trait / 方法 / 常量）在目标公共模块的依赖链里是否存在、签名是否一致。
+2. 上游可能已演进：基线 backend 已适配新接口，待对齐 backend 可能仍是旧签名 —— 以**当前上游接口**为准。
+3. **再次确认“净代码行数收益”没有被配套改动吃掉**：把共享类、backend 薄壳、pom/profile 变更一起算进去；若整体代码行数不降，停止抽取。
+4. **native/JNI 关联**：若 JVM 代码背后有 native（JNI `native` 方法），确认该 backend 的 cpp 侧是否真支持。
+ 反例教训：bolt 的 iceberg native 在 `cpp/bolt/CMakeLists.txt` 被注释禁用、底层库无 iceberg connector，
+ 此时 JVM 可对齐接口签名（纯声明不影响编译），但**不要**强行启用 native；如实告知用户该限制。
+
+## 第 3 步 — 选择抽取手法（按差异程度，从轻到重）
+
+1. **整体移动**（第一档，逐字一致）：把文件整体搬到公共模块，删除各 backend 重复副本。
+ 适用无 backend 符号依赖的通用类（如某 util / visitor / proto / transformer）。
+2. **抽象基类 + thin 子类**（第二档，主流手法）：
+ - 公共层放 `abstract class` / `trait`，封装共享逻辑，把后端差异点声明为**抽象方法**或**构造参数**。
+ - backend 留 `class XxxFactory extends SharedXxxFactory("backend-specific-const")` 这种壳，只传差异。
+ - 注意：基类若已 `extends` 某接口，子类**不要**重复 `with` 同一接口（Scala 会报错）。
+ - 注意：抽象基类构造器别加多余 `override val`（编译错误）。
+3. **共享测试基类**：测试 mixin/基类抽到公共层，把后端特定探测（如 `isNativeWritePlan`）留抽象方法。
+ 方法名要与各 backend 原调用名一致，避免改调用点。
+
+统一规则：
+- 公共文件**优先落 `backends-core`**；若属更底层通用能力且 backends-core 也依赖它，则落 `gluten-substrait`/`gluten-arrow`/`gluten-core`。
+- backend 侧**保留原接口名与调用链**，上层零感知。
+- profile 相关代码（celeborn / iceberg / uniffle）抽到公共模块时，公共模块也要配套加对应 profile（`src-celeborn` / `src-iceberg` 源根 + 依赖）。
+
+## 第 4 步 — 模块/构建配置（容易漏）
+
+- 新建公共模块时：在顶层 `pom.xml` 的 `<modules>` 注册；模块 `pom.xml` 配好 parent、scala 编译、provided 依赖。
+- 各 backend `pom.xml` 加对公共模块的 `compile` 依赖。
+- profile 专属代码：公共模块与 backend 都要在对应 profile 下加源根与依赖（如 celeborn 的 shaded client、iceberg 的 runtime + protoc）。
+- proto 抽到公共层后，**删除各 backend 重复的 `.proto`**，避免重复生成同名类冲突。
+
+## 第 5 步 — 工作流纪律（老规矩）
+
+- **一组一提交**：按功能分组（如 ①MetricsUpdater ②rewrite规则 ③execution glue ④JNI/datasource ⑤Iceberg ⑥Celeborn/Uniffle 工厂 ⑦测试基类），
+ 每组改完编译验证再提交，避免累积错误难定位。
+- **独立 worktree（硬性）**：必须从 `/data00/home/liyang.127/oap/gluten.dev` 的 `fake_main` 新建 worktree + 专属分支成型 patch，主仓只做对齐；如果当前工作目录不是从这个**指定仓库路径**与**指定基线分支**拉出的新 worktree，先停下迁移，不能就地抽取。这样可以避免后续 `cherry-pick` / `append` 时遇到“不是同一个对象库”的问题。
+- **范围克制**：用户只说抽 JVM 就不碰 native/其他；社区基线可能无私有 backend，私有 backend 的 thin 壳单独手做、不混进公共 patch。
+
+## 第 6 步 — 验证闭环（关键：分清 .m2 旧 jar 干扰）
+
+- **优先用 reactor 全量构建**（如 `make jar_spark35`，本质 `mvn -Pspark-3.5 package`），不要 install。
+- **`.m2` 旧 jar 陷阱**：单模块 `mvn -pl X -o` 或 reactor 解析依赖时，可能命中 `~/.m2` 里过期的
+ `gluten-*-SNAPSHOT.jar`，报“not a member / overrides nothing”等假错。判断方法：`javap` 查 target/classes
+ 含新符号但报错依旧 → 是 .m2 旧 jar 污染。处理：清理 `~/.m2/repository/org/apache/gluten/*/<version>` 后用 reactor 重编。
+- **区分 main vs test 编译**：`scala-compile-first`（main）通过即核心目标达成；`scala-test-compile-first`（test）
+ 的报错若是 backend 测试与上游 test-jar 不匹配的预存问题（如缺 `checkGlutenOperatorMatch`），需向用户确认是否在范围内。
+- 改完每组都编一次；style 失败用 `./dev/format-scala-code.sh` 或 `mvn spotless:apply` 修，别手抠空白。
+
+## 第 7 步 — 甄别“预存问题” vs “本次引入”
+
+编译报错时，先用 `git diff <add-backend-commit> -- <file> | wc -l` 判断该报错文件是否被本次改动触碰：
+- diff=0 且报错符号与本次抽取无关 → **预存问题**（backend 与上游基线脱节），如实告知用户、不擅自扩大范围。
+- 与本次抽取相关 → 按基线 backend 对齐修复。
+
+## 第 8 步 — 状态留痕
+
+把“抽了哪些 / 每组对应 commit / 暂缓哪些 / 为何暂缓 / native 限制”记入 `plan.md` 对应章节，便于跨 session 续作与回溯。
+
+## 一句话提炼
+
+**先从 `/data00/home/liyang.127/oap/gluten.dev` 的 `fake_main` 新建 worktree → 先算净代码行数收益（不能减少整体代码行数则不抽）→ 选基线 backend → 规范化 diff 分档（一致/可参数化/已分叉）→ 验证依赖方向(core 不依赖 backend)与接口/native 真实性 → 按差异从轻到重选手法（整体移动<抽象基类+thin子类）→ 公共落 backends-core、backend 留壳、配齐 pom/profile/删重复 proto → 一组一提交、reactor 构建验证(警惕 .m2 旧 jar)→ 甄别预存问题不越界 → 决策入档。** 风险与依赖判断永远先于代码技巧。
diff --git a/.github/skills/resolve-main-conflict/SKILL.md b/.github/skills/resolve-main-conflict/SKILL.md
new file mode 100644
index 0000000000..12a0110481
--- /dev/null
+++ b/.github/skills/resolve-main-conflict/SKILL.md
@@ -0,0 +1,104 @@
+---
+name: resolve-main-conflict
+description: >-
+ 当用户需要把 bolt 相关改动适配到演进后的上游基线（origin/main 或 liyang/fake_main）时，使用本技能。覆盖两类冲突：(1) git rebase/merge 产生的“显式冲突”；(2) 上游演进导致 bolt 代码/接口与上游不兼容的“隐式冲突”（无冲突标记但编译/语义已断）。不限语言、不限模块，可针对指定文件、目录或模块。强调“识别 old commit 里实质性的 bolt changes，按语义重新 apply 到新基线，零行为回退”。触发词：`/resolve-main-conflict`。
+argument-hint: "[ <file/dir/module ...> | (留空=全部受影响代码) ] 例: cpp/bolt/ 或 BoltSparkPlanExecApi.scala 或 shuffle"
+user-invocable: true
+---
+
+# 解决上游演进冲突 Skill
+
+把 bolt 相关改动安全地适配到演进后的上游基线（`origin/main` 或 `liyang/fake_main`）。
+**核心准则：既理解上游为什么改，也保留 bolt 为什么改，最终两边意图都不丢、行为不回退。**
+
+**硬性原则：凡处理过程中涉及“公共抽取 / 共享代码下沉”（例如把 bolt / velox 的公共 JVM 代码抽到 `backends-core`），必须从 `/data00/home/liyang.127/oap/gluten.dev` 的 `fake_main` 分支新建独立 git worktree + 专属分支实施；禁止直接在当前主仓 worktree、脏工作区、其它对象库 worktree，或任何非 `/data00/home/liyang.127/oap/gluten.dev` `fake_main` 派生的工作区里完成公共抽取。**
+
+本 Skill 必须处理**两类冲突**：
+- **显式冲突**：`git rebase` / `git merge` 报的、带 `<<<<<<< ======= >>>>>>>` 标记的冲突。
+- **隐式冲突**：没有冲突标记，但因上游（origin/main 或 liyang/fake_main）演进，
+ 导致 bolt 代码 / 接口与上游**不再兼容**（编译断裂、签名对不上、依赖符号被改名/删除/挪位、行为语义漂移）。
+ 这种最危险，rebase 会“成功”但产物是坏的。
+
+## 适用范围（必须支持）
+
+用户调用时可指定范围，本 Skill 必须遵守：
+- **某几个文件** / **某些目录** / **某个模块**（先 grep + 构建配置把模块映射到文件集）/ **留空**（全部受影响代码）。
+
+开始前先复述理解的范围与目标基线分支，避免越界。
+
+## 第 0 步 — 判别冲突类型并摸清现状（禁止盲改）
+
+1. 确认目标基线：`origin/main` 还是 `liyang/fake_main`？先 `git fetch`，
+ 用 `git log --oneline <base>..HEAD` 看本地（bolt）有哪些 commit，
+ 用 `git log --oneline HEAD..<base>` 看上游演进了什么。
+2. 判别类型：
+ - 有 `git status` 报 unmerged / 文件含冲突标记 → **显式冲突**，走第 1 步。
+ - 无冲突标记但担心 / 已知接口漂移 → **隐式冲突**，走第 2 步（务必主动排查，别等编译炸）。
+3. 三方视角（显式冲突时）：`git show :1:<file>`(base) / `:2:`(ours/bolt) / `:3:`(theirs/上游)，
+ 分别理解 bolt 意图与上游意图。
+
+## 第 1 步 — 处理显式冲突（缝合，不是二选一）
+
+对每个冲突块判断并处理：
+- **上游重构了签名/结构，bolt 新增了逻辑** → 在上游新结构上**重新挂载** bolt 逻辑，不要整段用 bolt 覆盖上游。
+- **两边改同一处但目的不同** → 合并两者意图；想不清就回三方 diff 找证据，必要时 AskUserQuestion 对齐。
+- **上游删除/移动了 bolt 依赖的符号** → 跟随上游新位置/新名字，不要把删掉的加回来。
+- **纯格式 vs 实质改动** → 取实质改动方，再套上游格式约定。
+
+红线：禁 `--ours`/`--theirs` 一把梭、禁 `reset --hard` 丢工作、禁 `--no-verify` 跳钩子。
+解冲突是“缝合”，不确定的实质语义差异要停下来问，别赌。
+
+## 第 2 步 — 处理隐式冲突（核心：抽 old commit 的实质 bolt changes，按语义重 apply）
+
+隐式冲突没有标记，要主动“考古 + 重放”。流程：
+
+1. **定位 bolt 的实质改动来源**：bolt 全量改动承载在固定的 old commit
+ **`d70e4d9cf616bc70a05ade97060a19072f43506`**（"add bolt backend in gluten"）。
+ 它是隐式冲突排查的权威基线——所有 bolt 特有改动都能在它里面找到原始形态。
+ 用 `git show d70e4d9cf6 -- <范围>` / `git log -p <base>..HEAD -- <范围>` 看清这些 commit
+ 对目标文件/接口做了什么。（若该 commit 不可达，再回退到 `git log -p <base>..HEAD` 兜底。）
+2. **区分“实质 bolt changes” vs “噪音”**：
+ - 实质：bolt 特有的新接口 / 新参数 / 新分支逻辑 / `#ifdef` 隔离 / 符号重命名适配 / 新增 backend 钩子。
+ - 噪音：纯格式、与上游重复的部分、已被上游以另一种方式实现的部分（这些**不要**再 apply）。
+3. **对照新基线现状**：把 old commit 里的实质改动，逐项和 `<base>` 上该文件的**当前**形态比对——
+ 上游可能已经改了签名/挪了位置/换了类型名。判断每个实质改动在新基线上**应该落在哪、怎么改写**。
+4. **按语义重新 apply（不是无脑 cherry-pick / patch -p1）**：
+ 在新基线的结构上把 bolt 实质改动重写挂载好（新参数加到上游新签名末尾、调用链跟随上游新顺序、
+ 符号跟随上游新命名）。目标是“bolt 的能力在新基线上等价存在”，而非“把旧代码贴回去”。
+5. **典型隐式断裂模式（重点排查）**：
+ - 上游给函数/构造器加了参数或改了顺序 → bolt 的调用点 / override 必须同步。
+ - 上游重命名/删除/移动了类型、命名空间、宏 → bolt 引用要跟随，缺失依赖要确认是否还存在。
+ - 上游新增了必须实现的抽象方法 → bolt 的子类要补实现。
+ - 某依赖在 bolt 侧根本不存在（如私有 backend 缺某 connector）→ 该改动不在 bolt apply，
+ 或用 `#ifdef` / 配置隔离，先确认引用是否本就是注释/不编译。
+
+## 第 3 步 — 收尾
+
+- 全局 grep 冲突标记 `<<<<<<<` / `=======` / `>>>>>>>`，必须为空。
+- 显式冲突：`git add` 后 `git rebase --continue` 或 `git commit`。
+- 隐式冲突：作为正常改动提交，commit message 写清“适配上游 X 演进：bolt 的 Y 能力重新挂载到新签名/新位置”。
+- 范围外仍有冲突而 rebase 必须整体推进时，明确告知用户，不要静默跳过。
+
+若在解冲突过程中决定进一步做“公共抽取 / 共享代码下沉”，先额外做一轮**净代码行数收益检查**：
+- 把“新增共享代码 + backend 薄壳 + 配套构建改动”的总行数，与抽取前各 backend 重复实现的总行数比较。
+- 若**不能减少整体代码行数**，则**不要把本次任务演进为公共抽取**；只做冲突修复/语义适配，不额外引入共享下沉。
+- 只有在减少重复、减少维护面且总代码量净下降时，才允许继续走公共抽取流程。
+
+## 第 4 步 — 验证闭环（隐式冲突尤其重要）
+
+目标：证明“适配后行为没回退、且确实编得过”。
+- **能编就先编**：窄范围增量编译/类型检查受影响文件，再扩到模块级；隐式冲突**必须**靠编译来兜底，
+ 因为它没有冲突标记提示你哪里断了。
+- 有测试跑相关测试；没有至少构建 + 静态检查。
+- 失败时定位是“缝合/重放错了”还是“上游还需 bolt 额外适配”，对症修，不要 retry 硬撞。
+
+## 第 5 步 — 复盘留痕
+
+向用户简述：上游这批演进改了什么、bolt 哪些实质改动被重新 apply、为何这样挂载、验证结果。
+多轮长任务把关键决策写入 `plan.md` 便于续作。
+
+若本次冲突处理演进为“顺手做公共抽取”，必须额外留痕说明：该抽取 patch 是不是从 `/data00/home/liyang.127/oap/gluten.dev` 的 `fake_main` 新建 worktree 起做的；若不是，先停下并迁移到符合要求的 worktree，再继续。这样可以避免后续把 patch append 回 `gluten.dev` 时遇到“不是同一个对象库”的问题。
+
+## 一句话提炼
+
+**先判别显式还是隐式 → 若要做公共抽取先切到从 `/data00/home/liyang.127/oap/gluten.dev` 的 `fake_main` 新建的 worktree，并先算净代码行数收益（不能减少整体代码行数则不抽）→ 显式：三方 diff 缝合不二选一；隐式：从 old commit 抽出实质 bolt changes、对照新基线按语义重新挂载（非无脑 patch）→ 清标记 → 编译/测试证零回退（隐式必须靠编译兜底）→ 复盘留痕。**
diff --git a/.gitignore b/.gitignore
index 85c0cd6f05..ceee520554 100644
--- a/.gitignore
+++ b/.gitignore
@@ -88,3 +88,7 @@ metastore_db/
# Claude Code local config
.claude/
+
+# Bolt backend build-generated files
+cpp/bolt/version/version.h
+cpp/gluten.conan.graph.html
diff --git a/.idea/vcs.xml b/.idea/vcs.xml
index f46f934cc9..2f596310db 100644
--- a/.idea/vcs.xml
+++ b/.idea/vcs.xml
@@ -1,30 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one or more
- ~ contributor license agreements. See the NOTICE file distributed with
- ~ this work for additional information regarding copyright ownership.
- ~ The ASF licenses this file to You under the Apache License, Version 2.0
- ~ (the "License"); you may not use this file except in compliance with
- ~ the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
<project version="4">
<component name="IssueNavigationConfiguration">
<option name="links">
<list>
<IssueNavigationLink>
<option name="issueRegexp" value="(?:#|GLUTEN-)(\d+)" />
- <!--
- GitHub share the sequence number of issues and pull requests, and it will redirect to
- the right place when the the sequence number not match kind.
- -->
<option name="linkRegexp" value="https://github.com/apache/gluten/issues/$1" />
</IssueNavigationLink>
</list>
@@ -32,5 +12,6 @@
</component>
<component name="VcsDirectoryMappings">
<mapping directory="$PROJECT_DIR$" vcs="Git" />
+ <mapping directory="$PROJECT_DIR$/ep/bolt" vcs="Git" />
</component>
-</project>
+</project>
\ No newline at end of file
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000000..838c6c8cbf
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
+BUILD_DIR := ${ROOT_DIR}/cpp/build
+CONAN_FILE_DIR := ${ROOT_DIR}/cpp/
+BUILD_TYPE=Debug
+ENABLE_ASAN ?= False
+LDB_BUILD ?= False
+BUILD_BENCHMARKS ?= False
+BUILD_TESTS ?= False
+BUILD_EXAMPLES ?= False
+BUILD_ORC ?= False
+ENABLE_PROTON ?= False
+
+# conan package info
+GLUTEN_BUILD_VERSION ?= main
+BOLT_BUILD_VERSION ?= main
+BUILD_USER ?=
+BUILD_CHANNEL ?=
+
+ENABLE_HDFS ?= True
+ENABLE_S3 ?= False
+RSS_PROFILE ?= ''
+
+ifeq ($(BUILD_BENCHMARKS),True)
+BUILD_ORC = True
+endif
+
+ARCH := $(shell arch)
+ifeq ($(ARCH), x86_64)
+ ARCH := amd64
+endif
+
+SHARED_LIBRARY ?= True
+
+# Gluten package version, read from package/pom.xml so jar paths stay in sync
+# when the project version bumps.
+GLUTEN_PACKAGE_VERSION := $(shell sed -n '0,/<version>\(.*\)<\/version>/s//\1/p' \
+ ${ROOT_DIR}/package/pom.xml | head -1)
+
+# Manually specify the number of bolt compilation threads by setting the BOLT_NUM_THREADS environment variable.
+# e.g. export BOLT_NUM_THREADS=50
+ifndef CI_NUM_THREADS
+ ifdef BOLT_NUM_THREADS
+ NUM_THREADS ?= $(BOLT_NUM_THREADS)
+ else
+ NUM_THREADS ?= $$(( $(shell grep -c ^processor /proc/cpuinfo) / 2 ))
+ endif
+else
+ NUM_THREADS ?= $(CI_NUM_THREADS)
+endif
+
+ALLOWED_VERSIONS := 11 17
+ifeq ($(JAVA_HOME),)
+ $(error ERROR: JAVA_HOME is not set)
+endif
+ifneq ($(wildcard $(JAVA_HOME)/bin/java),)
+ ifneq ($(wildcard $(JAVA_HOME)/bin/javac),)
+ JDK_VERSION := $(shell $(JAVA_HOME)/bin/java -version 2>&1 | sed -n 's/.*version "\(1\.\)\{0,1\}\([0-9]\+\).*/\2/p')
+ ifneq ($(filter $(JDK_VERSION),$(ALLOWED_VERSIONS)),$(JDK_VERSION))
+ $(error ERROR: JDK version $(JDK_VERSION) is not supported, only 11 and 17 are allowed now)
+ endif
+ endif
+endif
+
+.PHONY: build clean clean_cpp debug release RelWithDebInfo \
+ install_debug install_release release-with-tests debug-with-tests \
+ release-with-benchmarks debug-with-benchmarks \
+ release-with-tests-and-benchmarks debug-with-tests-and-benchmarks \
+ java jar jar-skip-check jar_spark33 jar_spark34 jar_spark35 \
+ fast-jar zip fast-zip test test_spark35 cpp-test-release cpp-test-debug
+
+bolt-recipe:
+ @echo "Install Bolt recipe into local cache"
+ rm -rf ep/bolt
+ git clone --depth=1 --branch ${BOLT_BUILD_VERSION} https://github.com/bytedance/bolt.git ep/bolt &&\
+ bash ep/bolt/scripts/install-bolt-deps.sh && \
+ conan export ep/bolt/conanfile.py --name=bolt --version=${BOLT_BUILD_VERSION} --user=${BUILD_USER} --channel=${BUILD_CHANNEL}
+ @echo "Bolt recipe has been installed"
+
+build:
+ mkdir -p ${BUILD_DIR} && mkdir -p ${BUILD_DIR}/releases &&\
+ cd ${CONAN_FILE_DIR} && export BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} &&\
+ ALL_CONAN_OPTIONS=" -o gluten/*:shared=${SHARED_LIBRARY} \
+ -o gluten/*:enable_hdfs=${ENABLE_HDFS} \
+ -o gluten/*:enable_s3=${ENABLE_S3} \
+ -o gluten/*:enable_asan=${ENABLE_ASAN} \
+ -o gluten/*:build_benchmarks=${BUILD_BENCHMARKS} \
+ -o gluten/*:build_tests=${BUILD_TESTS} \
+ -o gluten/*:build_examples=${BUILD_EXAMPLES} " && \
+ conan graph info . --name=gluten --version=${GLUTEN_BUILD_VERSION} --user=${BUILD_USER} --channel=${BUILD_CHANNEL} -c "arrow/*:tools.build:download_source=True" $${ALL_CONAN_OPTIONS} --format=html > gluten.conan.graph.html && \
+ NUM_THREADS=$(NUM_THREADS) conan install . --name=gluten --version=${GLUTEN_BUILD_VERSION} --user=${BUILD_USER} --channel=${BUILD_CHANNEL} \
+ -s llvm-core/*:build_type=Release -s build_type=${BUILD_TYPE} --build=missing $${ALL_CONAN_OPTIONS} && \
+ cmake --preset `echo conan-${BUILD_TYPE} | tr A-Z a-z` && \
+ cmake --build build/${BUILD_TYPE} -j $(NUM_THREADS) && \
+ if [ "${SHARED_LIBRARY}" = "True" ]; then cmake --build ${BUILD_DIR}/${BUILD_TYPE} --target install ; fi && \
+ if [ "${SHARED_LIBRARY}" = "False" ]; then \
+ conan export-pkg . --name=gluten --version=${GLUTEN_BUILD_VERSION} --user=${BUILD_USER} --channel=${BUILD_CHANNEL} -s build_type=${BUILD_TYPE} \
+ $${ALL_CONAN_OPTIONS} ; \
+ fi && cd -
+
+release :
+ $(MAKE) build BUILD_TYPE=Release GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL}
+
+debug:
+ $(MAKE) build BUILD_TYPE=Debug GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL}
+
+RelWithDebInfo:
+ $(MAKE) build BUILD_TYPE=RelWithDebInfo GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL}
+
+clean_cpp:
+ rm -rf ${ROOT_DIR}/cpp/build &&\
+ rm -f cpp/conan.lock cpp/conaninfo.txt cpp/graph_info.json CMakeCache.txt
+
+install_debug:
+ $(MAKE) clean_cpp
+ $(MAKE) debug SHARED_LIBRARY=False
+
+install_release:
+ $(MAKE) clean_cpp
+ $(MAKE) release SHARED_LIBRARY=False
+
+release-with-tests :
+ $(MAKE) build BUILD_TYPE=Release GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL} BUILD_TESTS=True
+
+debug-with-tests :
+ $(MAKE) build BUILD_TYPE=Debug GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL} BUILD_TESTS=True
+
+release-with-benchmarks :
+ $(MAKE) build BUILD_TYPE=Release GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} B UILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL} BUILD_BENCHMARKS=True
+
+debug-with-benchmarks :
+ $(MAKE) build BUILD_TYPE=Debug GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL} BUILD_BENCHMARKS=True
+
+release-with-tests-and-benchmarks :
+ $(MAKE) build BUILD_TYPE=Release GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL} BUILD_BENCHMARKS=True BUILD_TESTS=True
+
+debug-with-tests-and-benchmarks :
+ $(MAKE) build BUILD_TYPE=Debug GLUTEN_BUILD_VERSION=${GLUTEN_BUILD_VERSION} BOLT_BUILD_VERSION=${BOLT_BUILD_VERSION} BUILD_USER=${BUILD_USER} BUILD_CHANNEL=${BUILD_CHANNEL} BUILD_BENCHMARKS=True BUILD_TESTS=True
+
+arrow:
+ bash dev/build_bolt_arrow.sh
+
+# build gluten jar
+jar:
+ java -version && mvn package -Pbackends-bolt -Pspark-3.3 -Pceleborn -DskipTests -Denforcer.skip=true -Pjava-8 -Ppaimon &&\
+ mkdir -p output && \
+ rm -rf output/gluten-spark*.jar
+ mv package/target/gluten-package-$(GLUTEN_PACKAGE_VERSION).jar output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+jar-skip-check:
+ java -version && mvn package -Pbackends-bolt -Pspark-3.2 -Pceleborn -DskipTests -Denforcer.skip=true -Pjava-8 -Ppaimon -Dcheckstyle.skip=true -Dspotless.check.skip=true &&\
+ mkdir -p output && \
+ rm -rf output/gluten-spark*.jar
+ mv package/target/gluten-package-$(GLUTEN_PACKAGE_VERSION).jar output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+spark32-las:
+ java -version && mvn package -Pbackends-bolt -Pspark-3.2-las -Pceleborn -DskipTests -Denforcer.skip=true -Pjava-8 -Ppaimon &&\
+ mkdir -p output && \
+ rm -rf output/gluten-spark*.jar
+ mv package/target/gluten-package-$(GLUTEN_PACKAGE_VERSION).jar output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+fast-jar:
+ if [ ! -f "output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar" ] ; then \
+ $(MAKE) jar; \
+ else \
+ jar uf output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar -C cpp/build/releases/ libbolt_backend.so; \
+ fi
+
+zip:
+ $(MAKE) jar
+ rm -rf output/gluten-spark*.zip
+ zip -j output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.zip output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+fast-zip:
+ $(MAKE) fast-jar
+ rm -rf output/gluten-spark*.zip
+ zip -j output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.zip output/gluten-spark3.2_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+jar_spark33:
+ java -version && mvn -T32 clean package -Pbackends-bolt -Pspark-3.3 -Pceleborn -Piceberg -DskipTests -Denforcer.skip=true -Ppaimon && \
+ mkdir -p output && \
+ rm -rf output/gluten-spark*.jar
+ mv package/target/gluten-package-$(GLUTEN_PACKAGE_VERSION).jar output/gluten-spark3.3_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+jar_spark34:
+ java -version && mvn clean package -Pbackends-bolt -Pspark-3.4 -Pceleborn -Piceberg -DskipTests -Denforcer.skip=true -Ppaimon && \
+ mkdir -p output && \
+ rm -rf output/gluten-spark*.jar
+ mv package/target/gluten-package-$(GLUTEN_PACKAGE_VERSION).jar output/gluten-spark3.4_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+jar_spark35:
+ java -version && mvn -T32 package -Pbackends-bolt -Pspark-3.5 -Phadoop-3.2 -Pceleborn -Piceberg -DskipTests -Denforcer.skip=true && \
+ mkdir -p output && \
+ rm -rf output/gluten-spark*.jar
+ mv package/target/gluten-package-$(GLUTEN_PACKAGE_VERSION).jar output/gluten-spark3.5_2.12-1.0.0-SNAPSHOT-jar-with-dependencies.jar
+
+test:
+ mvn -Pbackends-bolt -Pspark-3.2 -Pceleborn -Ppaimon package -Denforcer.skip=true
+
+test_spark35:
+ mvn -Pbackends-bolt -Pspark-3.5 -Ppaimon -Phadoop-3.2 -Pceleborn -Piceberg package -Denforcer.skip=true
+
+cpp-test-release: release-with-tests
+ cd $(BUILD_DIR)/Release && ctest --timeout 7200 -j $(NUM_THREADS) --output-on-failure -V
+
+cpp-test-debug: debug-with-tests
+ cd $(BUILD_DIR)/Debug && ctest --timeout 7200 -j $(NUM_THREADS) --output-on-failure -V
+
+clean :
+ $(MAKE) clean_cpp
+ mvn clean -Pbackends-bolt -Pspark-3.2 -Pceleborn -Ppaimon -DskipTests -Denforcer.skip=true && \
+ rm -rf ${ROOT_DIR}/output/gluten-*.jar
diff --git a/README.md b/README.md
index ea54568d33..dc73185cdf 100644
--- a/README.md
+++ b/README.md
@@ -154,6 +154,60 @@ ClickHouse backend demonstrated an average speedup of 2.12x, with up to 3.48x sp
<sub>Test environment: a 8-nodes AWS cluster with 1TB data, using Spark 3.1.1 as the baseline and with Gluten integrated into the same Spark version.</sub>
+### Bolt Backend
+#### Prerequisites
+* Linux operating system
+* GCC 10/11/12 or Clang 16
+* python 3 (virtualenv or Conda) for conan
+
+Linux with kernel version(>5.4) is preferred, since Bolt will enable io-uring when the kernel supports.
+
+if the system gcc version is too older, it is recommended to install GCC from source code:
+```shell
+# run with root privilege
+bash ./dev/install-gcc.sh 12.5.0
+```
+
+Bolt adopts Conan as its package manager. Conan is an open-source, cross-platform package management tool.
+We provide dedicated scripts to assist developers in setting up and installing Bolt's dependencies.
+```shell
+bash ./dev/install-conan.sh
+```
+
+We also provide a Dockerfile to build a Docker image for the **Bolt** backend, it includes all the prerequisites required to build Gluten with Bolt backend.
+```shell
+docker buildx build -t bolt -f dev/docker/Dockerfile.centos8-bolt .
+```
+
+#### Build Bolt Backend
+To install bolt recipe for Gluten:
+```shell
+# Install the recipes of Bolt and its third-party dependencies
+make bolt-recipe
+
+# specific a version of Bolt (release or branch)
+# `main` branch is the default
+make bolt-recipe BOLT_BUILD_VERSION=main
+```
+
+To build bolt backend:
+```shell
+make release
+
+# or specific the version for Bolt, and the version for Gluten
+make release BOLT_BUILD_VERSION=main GLUTEN_BUILD_VERSION=main
+```
+Note that, the missing third-parties binaries will be built from source for the first time.
+
+To build gluten:
+
+```shell
+# install arrow dependency for gluten
+make arrow
+
+make jar_spark35
+```
+
## 8. Qualification Tool
The [Qualification Tool](./tools/qualification-tool/README.md) is a utility to analyze Spark event log files and assess the compatibility and performance of SQL workloads with Gluten. This tool helps users understand how their workloads can benefit from Gluten.
diff --git a/backends-bolt/benchmark/ColumnarTableCacheBenchmark-results.txt b/backends-bolt/benchmark/ColumnarTableCacheBenchmark-results.txt
new file mode 100644
index 0000000000..43e1faa7cc
--- /dev/null
+++ b/backends-bolt/benchmark/ColumnarTableCacheBenchmark-results.txt
@@ -0,0 +1,23 @@
+OpenJDK 64-Bit Server VM 1.8.0_322-b06 on Mac OS X 13.5
+Apple M1 Pro
+table cache count: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+disable columnar table cache 16773 17024 401 1.2 838.7 1.0X
+enable columnar table cache 9985 10051 65 2.0 499.3 1.0X
+
+
+OpenJDK 64-Bit Server VM 1.8.0_322-b06 on Mac OS X 13.5
+Apple M1 Pro
+table cache column pruning: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+disable columnar table cache 16429 16873 688 1.2 821.5 1.0X
+enable columnar table cache 15118 15495 456 1.3 755.9 1.0X
+
+
+OpenJDK 64-Bit Server VM 1.8.0_322-b06 on Mac OS X 13.5
+Apple M1 Pro
+table cache filter: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+disable columnar table cache 22895 23527 722 0.9 1144.7 1.0X
+enable columnar table cache 16673 17462 765 1.2 833.7 1.0X
+
diff --git a/backends-bolt/pom.xml b/backends-bolt/pom.xml
new file mode 100755
index 0000000000..912c284359
--- /dev/null
+++ b/backends-bolt/pom.xml
@@ -0,0 +1,540 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-parent</artifactId>
+ <version>1.8.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>backends-bolt</artifactId>
+ <packaging>jar</packaging>
+ <name>Gluten Backends Bolt</name>
+
+ <properties>
+ <cpp.build.dir>../cpp/build/</cpp.build.dir>
+ <cpp.releases.dir>${cpp.build.dir}/releases/</cpp.releases.dir>
+ <byte-buddy.version>1.9.3</byte-buddy.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-substrait</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>backends-core</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>backends-core</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ <version>${protobuf.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-substrait</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-network-common_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-core</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-arrow</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-hive_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scalacheck</groupId>
+ <artifactId>scalacheck_${scala.binary.version}</artifactId>
+ <version>1.17.0</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>${scala.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scala-lang.modules</groupId>
+ <artifactId>scala-collection-compat_${scala.binary.version}</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ <version>2.23.4</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <version>${byte-buddy.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.scalatestplus</groupId>
+ <artifactId>scalatestplus-mockito_${scala.binary.version}</artifactId>
+ <version>1.0.0-M2</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.scalatestplus</groupId>
+ <artifactId>scalatestplus-scalacheck_${scala.binary.version}</artifactId>
+ <version>3.1.0.0-RC2</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <version>2.14.0</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-hive_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <!-- Fasterxml -->
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.module</groupId>
+ <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.google.jimfs</groupId>
+ <artifactId>jimfs</artifactId>
+ <version>1.3.0</version>
+ <scope>compile</scope>
+ </dependency>
+ <!-- Java Faker for generating random data -->
+ <dependency>
+ <groupId>com.github.javafaker</groupId>
+ <artifactId>javafaker</artifactId>
+ <version>1.0.2</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.vladsch.flexmark</groupId>
+ <artifactId>flexmark-all</artifactId>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <resources>
+ <resource>
+ <directory>${project.basedir}/src/main/resources</directory>
+ </resource>
+ <resource>
+ <targetPath>${platform}/${arch}</targetPath>
+ <directory>${cpp.releases.dir}</directory>
+ </resource>
+ </resources>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-resources-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>net.alchim31.maven</groupId>
+ <artifactId>scala-maven-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.scalastyle</groupId>
+ <artifactId>scalastyle-maven-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>com.diffplug.spotless</groupId>
+ <artifactId>spotless-maven-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <junitxml>.</junitxml>
+ <tagsToExclude>${tagsToExclude}</tagsToExclude>
+ <systemProperties>
+ <bolt.udf.lib.path>${cpp.build.dir}/bolt/udf/examples/libmyudf.so,${cpp.build.dir}/bolt/udf/examples/libmyudaf.so</bolt.udf.lib.path>
+ </systemProperties>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-test-jar</id>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ <phase>test-compile</phase>
+ </execution>
+ </executions>
+ </plugin>
+ <!-- compile proto buffer files using copied protoc binary -->
+ <plugin>
+ <groupId>org.xolstice.maven.plugins</groupId>
+ <artifactId>protobuf-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>compile-gluten-proto</id>
+ <goals>
+ <goal>compile</goal>
+ <goal>test-compile</goal>
+ </goals>
+ <phase>generate-sources</phase>
+ <configuration>
+ <protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
+ <protoSourceRoot>src/main/resources/org/apache/gluten/proto</protoSourceRoot>
+ <clearOutputDirectory>false</clearOutputDirectory>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
+ <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+ </build>
+
+ <profiles>
+ <profile>
+ <id>exclude-tests</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <tagsToExclude>org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.spark.tags.SkipTest</tagsToExclude>
+ </properties>
+ </profile>
+ <profile>
+ <id>celeborn</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-celeborn</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.celeborn</groupId>
+ <artifactId>celeborn-client-spark-${spark.major.version}-shaded_${scala.binary.version}</artifactId>
+ <version>${celeborn.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.celeborn</groupId>
+ <artifactId>celeborn-client-spark-${spark.major.version}_${scala.binary.version}</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.celeborn</groupId>
+ <artifactId>celeborn-spark-${spark.major.version}-columnar-shuffle_${scala.binary.version}</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>uniffle</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-uniffle</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.uniffle</groupId>
+ <artifactId>rss-client-spark${spark.major.version}-shaded</artifactId>
+ <version>${uniffle.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>iceberg</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <properties>
+ <!-- The version should be same with the version in assertj-core -->
+ <byte-buddy.version>1.14.18</byte-buddy.version>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-iceberg</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-iceberg</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-spark-runtime-${sparkbundle.version}_${scala.binary.version}</artifactId>
+ <version>${iceberg.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-spark-${sparkbundle.version}_${scala.binary.version}</artifactId>
+ <version>${iceberg.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-hadoop</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-hive-metastore</artifactId>
+ <version>${iceberg.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-api</artifactId>
+ <version>${iceberg.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-data</artifactId>
+ <version>${iceberg.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-avro</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-spark-extensions-${sparkbundle.version}_${scala.binary.version}</artifactId>
+ <version>${iceberg.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.assertj</groupId>
+ <artifactId>assertj-core</artifactId>
+ <version>3.26.3</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.13.2</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ <version>5.11.4</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.awaitility</groupId>
+ <artifactId>awaitility</artifactId>
+ <version>4.2.2</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>delta</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-delta</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-delta</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>io.delta</groupId>
+ <artifactId>${delta.package.name}_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>hudi</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-hudi</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-hudi</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark${sparkbundle.version}-bundle_${scala.binary.version}</artifactId>
+ <version>${hudi.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>paimon</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-paimon</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-paimon</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.paimon</groupId>
+ <artifactId>paimon-spark-${sparkbundle.version}_${scala.binary.version}</artifactId>
+ <version>${paimon.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+ </profiles>
+</project>
diff --git a/backends-bolt/src-celeborn/main/java/org/apache/gluten/vectorized/CelebornPartitionWriterJniWrapper.java b/backends-bolt/src-celeborn/main/java/org/apache/gluten/vectorized/CelebornPartitionWriterJniWrapper.java
new file mode 100644
index 0000000000..7cbfe68955
--- /dev/null
+++ b/backends-bolt/src-celeborn/main/java/org/apache/gluten/vectorized/CelebornPartitionWriterJniWrapper.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.vectorized;
+
+import org.apache.gluten.runtime.Runtime;
+import org.apache.gluten.runtime.RuntimeAware;
+
+public class CelebornPartitionWriterJniWrapper implements RuntimeAware {
+ private final Runtime runtime;
+
+ private CelebornPartitionWriterJniWrapper(org.apache.gluten.runtime.Runtime runtime) {
+ this.runtime = runtime;
+ }
+
+ public static CelebornPartitionWriterJniWrapper create(Runtime runtime) {
+ return new CelebornPartitionWriterJniWrapper(runtime);
+ }
+
+ @Override
+ public long rtHandle() {
+ return runtime.getHandle();
+ }
+
+ public native long createPartitionWriter(
+ int numPartitions,
+ String codec,
+ String codecBackend,
+ int compressionLevel,
+ int compressionBufferSize,
+ int pushBufferMaxSize,
+ long sortBufferMaxSize,
+ Object pusher);
+}
diff --git a/backends-bolt/src-celeborn/main/resources/META-INF/services/org.apache.spark.shuffle.gluten.celeborn.CelebornColumnarBatchSerializerFactory b/backends-bolt/src-celeborn/main/resources/META-INF/services/org.apache.spark.shuffle.gluten.celeborn.CelebornColumnarBatchSerializerFactory
new file mode 100644
index 0000000000..c31eafd597
--- /dev/null
+++ b/backends-bolt/src-celeborn/main/resources/META-INF/services/org.apache.spark.shuffle.gluten.celeborn.CelebornColumnarBatchSerializerFactory
@@ -0,0 +1 @@
+org.apache.spark.shuffle.BoltCelebornColumnarBatchSerializerFactory
diff --git a/backends-bolt/src-celeborn/main/resources/META-INF/services/org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleWriterFactory b/backends-bolt/src-celeborn/main/resources/META-INF/services/org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleWriterFactory
new file mode 100644
index 0000000000..af4f7806d2
--- /dev/null
+++ b/backends-bolt/src-celeborn/main/resources/META-INF/services/org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleWriterFactory
@@ -0,0 +1 @@
+org.apache.spark.shuffle.BoltCelebornColumnarShuffleWriterFactory
diff --git a/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarBatchSerializer.scala b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarBatchSerializer.scala
new file mode 100644
index 0000000000..7046ea9511
--- /dev/null
+++ b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarBatchSerializer.scala
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.config.{BoltConfig, GlutenConfig}
+import org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators
+import org.apache.gluten.proto.ShuffleReaderInfo
+import org.apache.gluten.runtime.Runtimes
+import org.apache.gluten.shuffle.{BoltShuffleReaderJniWrapper, BoltShuffleReaderMetrics}
+import org.apache.gluten.utils.ArrowAbiUtil
+import org.apache.gluten.vectorized._
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.SHUFFLE_COMPRESS
+import org.apache.spark.serializer.{DeserializationStream, SerializationStream, SerializerInstance}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.utils.SparkSchemaUtil
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.task.{TaskResource, TaskResources}
+
+import org.apache.arrow.c.ArrowSchema
+import org.apache.arrow.memory.BufferAllocator
+import org.apache.celeborn.client.read.CelebornInputStream
+
+import java.io._
+import java.nio.ByteBuffer
+import java.util.UUID
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.reflect.ClassTag
+
+class CelebornColumnarBatchSerializer(
+ schema: StructType,
+ readBatchNumRows: SQLMetric,
+ numOutputRows: SQLMetric,
+ decompressTime: SQLMetric,
+ deserializeTime: SQLMetric,
+ totalReadTime: SQLMetric)
+ extends SettableColumnarBatchSerializer(
+ readBatchNumRows,
+ numOutputRows,
+ decompressTime,
+ deserializeTime,
+ totalReadTime)
+ with Serializable {
+
+ /** Creates a new [[SerializerInstance]]. */
+ override def newInstance(): SerializerInstance = {
+ new CelebornColumnarBatchSerializerInstance(
+ schema,
+ readBatchNumRows,
+ numOutputRows,
+ decompressTime,
+ deserializeTime,
+ totalReadTime,
+ numPartitions,
+ partitionShortName)
+ }
+}
+
+private class CelebornColumnarBatchSerializerInstance(
+ schema: StructType,
+ readBatchNumRows: SQLMetric,
+ numOutputRows: SQLMetric,
+ decompressTime: SQLMetric,
+ deserializeTime: SQLMetric,
+ totalReadTime: SQLMetric,
+ numPartitions: Int,
+ partitionShortName: String)
+ extends SerializerInstance
+ with Logging {
+
+ private val runtime =
+ Runtimes.contextInstance(BackendsApiManager.getBackendName, "CelebornShuffleReader")
+
+ private val shuffleReaderHandle = {
+ val allocator: BufferAllocator = ArrowBufferAllocators
+ .contextInstance(classOf[CelebornColumnarBatchSerializerInstance].getSimpleName)
+ .newChildAllocator("GlutenColumnarBatch deserialize", 0, Long.MaxValue)
+ val arrowSchema =
+ SparkSchemaUtil.toArrowSchema(schema, SQLConf.get.sessionLocalTimeZone)
+ val cSchema = ArrowSchema.allocateNew(allocator)
+ ArrowAbiUtil.exportSchema(allocator, arrowSchema, cSchema)
+ val conf = SparkEnv.get.conf
+ val compressionCodec =
+ if (conf.getBoolean(SHUFFLE_COMPRESS.key, SHUFFLE_COMPRESS.defaultValue.get)) {
+ GlutenShuffleUtils.getCompressionCodec(conf)
+ } else {
+ null // uncompressed
+ }
+ val compressionCodecBackend =
+ GlutenConfig.get.columnarShuffleCodecBackend.orNull
+ val jniWrapper = BoltShuffleReaderJniWrapper.create(runtime)
+ val batchSize = GlutenConfig.get.maxBatchSize
+
+ val shuffleBatchByteSize = BoltConfig.get.maxShuffleBatchByteSize
+ val forceShuffleWriterType = BoltConfig.get.forceShuffleWriterType
+ val builder = ShuffleReaderInfo.newBuilder()
+ builder
+ .setBatchSize(batchSize)
+ .setShuffleBatchByteSize(shuffleBatchByteSize)
+ .setNumPartitions(numPartitions)
+ .setPartitionShortName(partitionShortName)
+ .setForcedWriterType(forceShuffleWriterType)
+ .setCompressionType(compressionCodec)
+ .setCodec(compressionCodecBackend)
+ val handle = jniWrapper
+ .make(
+ cSchema.memoryAddress(),
+ builder.build().toByteArray
+ )
+ // Close shuffle reader instance as lately as the end of task processing,
+ // since the native reader could hold a reference to memory pool that
+ // was used to create all buffers read from shuffle reader. The pool
+ // should keep alive before all buffers to finish consuming.
+ TaskResources.addRecycler(s"CelebornShuffleReaderHandle_$handle", 50) {
+ // Collect Metrics
+ val readerMetrics = new BoltShuffleReaderMetrics()
+ jniWrapper.populateMetrics(handle, readerMetrics)
+ decompressTime += readerMetrics.getDecompressTime
+ deserializeTime += readerMetrics.getDeserializeTime
+ jniWrapper.close(handle)
+ cSchema.release()
+ cSchema.close()
+ allocator.close()
+ }
+ handle
+ }
+
+ override def deserializeStream(in: InputStream): DeserializationStream = {
+ val startTime = System.nanoTime()
+ val r = new TaskDeserializationStream(in)
+ totalReadTime += (System.nanoTime() - startTime)
+ r
+ }
+
+ private class TaskDeserializationStream(in: InputStream)
+ extends DeserializationStream
+ with TaskResource {
+ private val streamReader = ShuffleStreamReader(Iterator((null, in)))
+
+ private var wrappedOut: ColumnarBatchOutIterator = _
+
+ private var cb: ColumnarBatch = _
+
+ private var numBatchesTotal: Long = _
+ private var numRowsTotal: Long = _
+
+ private val isEmptyStream: Boolean = in.equals(CelebornInputStream.empty())
+
+ // Otherwise calling close() twice would cause resource ID not found error.
+ private val closeCalled: AtomicBoolean = new AtomicBoolean(false)
+
+ // Otherwise calling release() twice would cause #close0() to be called twice.
+ private val releaseCalled: AtomicBoolean = new AtomicBoolean(false)
+
+ private val resourceId = UUID.randomUUID().toString
+
+ TaskResources.addResource(resourceId, this)
+
+ override def asKeyValueIterator: Iterator[(Any, Any)] = new Iterator[(Any, Any)] {
+ private var gotNext = false
+ private var nextValue: (Any, Any) = _
+ private var finished = false
+
+ def getNext: (Any, Any) = {
+ try {
+ (readKey[Any](), readValue[Any]())
+ } catch {
+ case eof: EOFException =>
+ finished = true
+ null
+ }
+ }
+
+ override def hasNext: Boolean = {
+ if (!isEmptyStream && !finished) {
+ if (!gotNext) {
+ nextValue = getNext
+ gotNext = true
+ }
+ }
+ val hasNext = !isEmptyStream && !finished
+ if (!hasNext) {
+ TaskDeserializationStream.this.close()
+ }
+ hasNext
+ }
+
+ override def next(): (Any, Any) = {
+ if (!hasNext) {
+ throw new NoSuchElementException("End of stream")
+ }
+ gotNext = false
+ nextValue
+ }
+ }
+
+ override def asIterator: Iterator[Any] = {
+ // This method is never called by shuffle code.
+ throw new UnsupportedOperationException
+ }
+
+ override def readKey[T: ClassTag](): T = {
+ // We skipped serialization of the key in writeKey(), so just return a dummy value since
+ // this is going to be discarded anyways.
+ null.asInstanceOf[T]
+ }
+
+ @throws(classOf[EOFException])
+ override def readValue[T: ClassTag](): T = {
+ val startTime = System.nanoTime()
+ initStream()
+ if (cb != null) {
+ cb.close()
+ cb = null
+ }
+ val batch = {
+ val maybeBatch =
+ try {
+ wrappedOut.next()
+ } catch {
+ case ioe: IOException =>
+ this.close()
+ logError("Failed to load next RecordBatch", ioe)
+ throw ioe
+ }
+ if (maybeBatch == null) {
+ // EOF reached
+ this.close()
+ totalReadTime += (System.nanoTime() - startTime)
+ throw new EOFException
+ }
+ maybeBatch
+ }
+ totalReadTime += (System.nanoTime() - startTime)
+ val numRows = batch.numRows()
+ logDebug(s"Read ColumnarBatch of $numRows rows")
+ numBatchesTotal += 1
+ numRowsTotal += numRows
+ cb = batch
+ cb.asInstanceOf[T]
+ }
+
+ override def readObject[T: ClassTag](): T = {
+ // This method is never called by shuffle code.
+ throw new UnsupportedOperationException
+ }
+
+ override def close(): Unit = {
+ if (!closeCalled.compareAndSet(false, true)) {
+ return
+ }
+ // Would remove the resource object from registry to lower GC pressure.
+ TaskResources.releaseResource(resourceId)
+ }
+
+ override def release(): Unit = {
+ if (!releaseCalled.compareAndSet(false, true)) {
+ return
+ }
+ close0()
+ }
+
+ private def close0(): Unit = {
+ if (numBatchesTotal > 0) {
+ readBatchNumRows.set(numRowsTotal.toDouble / numBatchesTotal)
+ }
+ numOutputRows += numRowsTotal
+ if (wrappedOut != null) {
+ wrappedOut.close()
+ }
+ streamReader.close()
+ if (cb != null) {
+ cb.close()
+ }
+ }
+
+ private def initStream(): Unit = {
+ if (wrappedOut == null) {
+ wrappedOut = new ColumnarBatchOutIterator(
+ runtime,
+ ShuffleReaderJniWrapper
+ .create(runtime)
+ .read(shuffleReaderHandle, streamReader))
+ }
+ }
+
+ override def resourceName(): String = getClass.getName
+ }
+
+ // Columnar shuffle write process don't need this.
+ override def serializeStream(s: OutputStream): SerializationStream =
+ throw new UnsupportedOperationException
+
+ // These methods are never called by shuffle code.
+ override def serialize[T: ClassTag](t: T): ByteBuffer = throw new UnsupportedOperationException
+
+ override def deserialize[T: ClassTag](bytes: ByteBuffer): T =
+ throw new UnsupportedOperationException
+
+ override def deserialize[T: ClassTag](bytes: ByteBuffer, loader: ClassLoader): T =
+ throw new UnsupportedOperationException
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarBatchSerializerFactory.scala
similarity index 79%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarBatchSerializerFactory.scala
index 632c806075..0b7ded89f4 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarBatchSerializerFactory.scala
@@ -14,13 +14,8 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.spark.shuffle
-#pragma once
-
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+class BoltCelebornColumnarBatchSerializerFactory
+ extends SharedCelebornColumnarBatchSerializerFactory(
+ "org.apache.spark.shuffle.CelebornColumnarBatchSerializer")
diff --git a/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarShuffleWriter.scala b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarShuffleWriter.scala
new file mode 100644
index 0000000000..4a96d4d71a
--- /dev/null
+++ b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarShuffleWriter.scala
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.shuffle
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.backendsapi.bolt.WholeStageIteratorWrapper
+import org.apache.gluten.columnarbatch.ColumnarBatches
+import org.apache.gluten.config.BoltConfig
+import org.apache.gluten.memory.memtarget.{MemoryTarget, Spiller}
+import org.apache.gluten.proto.{ShuffleWriterInfo, ShuffleWriterResult}
+import org.apache.gluten.runtime.Runtimes
+import org.apache.gluten.shuffle.{BoltShuffleWriterJniWrapper, BoltSplitResult}
+
+import org.apache.spark._
+import org.apache.spark.memory.SparkMemoryUtil
+import org.apache.spark.scheduler.MapStatus
+import org.apache.spark.shuffle.celeborn.CelebornShuffleHandle
+import org.apache.spark.sql.vectorized.ColumnarBatch
+import org.apache.spark.util.SparkResourceUtil
+
+import org.apache.celeborn.client.ShuffleClient
+import org.apache.celeborn.common.CelebornConf
+
+import java.io.IOException
+
+import scala.collection.JavaConverters._
+
+class BoltCelebornColumnarShuffleWriter[K, V](
+ shuffleId: Int,
+ handle: CelebornShuffleHandle[K, V, V],
+ context: TaskContext,
+ celebornConf: CelebornConf,
+ client: ShuffleClient,
+ writeMetrics: ShuffleWriteMetricsReporter)
+ extends CelebornColumnarShuffleWriter[K, V](
+ shuffleId,
+ handle,
+ context,
+ celebornConf,
+ client,
+ writeMetrics) {
+ private val runtime =
+ Runtimes.contextInstance(BackendsApiManager.getBackendName, "CelebornShuffleWriter")
+
+ private val forceShuffleWriterType =
+ BoltConfig.get.forceShuffleWriterType
+
+ private val useV2PreAllocSizeThreshold =
+ BoltConfig.get.useV2PreallocSizeThreshold
+
+ private val rowVectorModeCompressionMinColumns =
+ BoltConfig.get.rowVectorModeCompressionMinColumns
+
+ private val rowVectorModeCompressionMaxBufferSize =
+ BoltConfig.get.rowvectorModeCompressionMaxBufferSize
+
+ private val accumulateBatchMaxColumns =
+ BoltConfig.get.accumulateBatchMaxColumns
+
+ private val accumulateBatchMaxBatches =
+ BoltConfig.get.accumulateBatchMaxBatches
+
+ private val recommendedColumn2RowSize =
+ BoltConfig.get.recommendedColumn2RowSize
+
+ private val shuffleCheckRatio =
+ BoltConfig.get.shuffleCheckRatio
+
+ private val enableVectorCombination =
+ BoltConfig.get.enableVectorCombination
+
+ private val shuffleWriterJniWrapper = BoltShuffleWriterJniWrapper.create(runtime)
+
+ private var splitResult: BoltSplitResult = _
+
+ private def availableOffHeapPerTask(): Long = {
+ SparkMemoryUtil.getCurrentAvailableOffHeapMemory / SparkResourceUtil.getTaskSlots(conf)
+ }
+
+ private def getShuffleWriterInfo(): ShuffleWriterInfo = {
+ val builder = ShuffleWriterInfo.newBuilder()
+ builder.setPartitioningName(dep.nativePartitioning.getShortName)
+ builder.setNumPartitions(dep.nativePartitioning.getNumPartitions)
+ builder.setStartPartitionId(
+ GlutenShuffleUtils.getStartPartitionId(dep.nativePartitioning, context.partitionId))
+ builder.setTaskAttemptId(context.taskAttemptId())
+ builder.setBufferSize(nativeBufferSize)
+ builder.setMergeBufferSize(0)
+ builder.setMergeThreshold(0)
+ builder.setCompressionCodec(compressionCodec.orNull)
+ builder.setCompressionBackend("none")
+ builder.setCompressionLevel(compressionLevel)
+ builder.setCompressionThreshold(BoltConfig.get.columnarShuffleCompressionThreshold)
+ builder.setCompressionMode(BoltConfig.get.columnarShuffleCompressionMode)
+
+ builder.setDataFile("")
+ builder.setNumSubDirs(0)
+ builder.setLocalDirs("")
+ builder.setReallocThreshold(BoltConfig.get.columnarShuffleReallocThreshold)
+ builder.setMemLimit(availableOffHeapPerTask())
+ builder.setPushBufferMaxSize(clientPushBufferMaxSize)
+ builder.setShuffleBatchByteSize(BoltConfig.get.maxShuffleBatchByteSize)
+ builder.setWriterType("celeborn")
+ builder.setForcedWriterType(forceShuffleWriterType)
+ builder.setUseV2PreallocThreshold(useV2PreAllocSizeThreshold)
+ builder.setRowCompressionMinCols(rowVectorModeCompressionMinColumns)
+ builder.setRowCompressionMaxBuffer(rowVectorModeCompressionMaxBufferSize)
+ builder.setEnableVectorCombination(enableVectorCombination)
+ builder.setAccumulateBatchMaxColumns(accumulateBatchMaxColumns)
+ builder.setAccumulateBatchMaxBatches(accumulateBatchMaxBatches)
+ builder.setRecommendedC2RSize(recommendedColumn2RowSize)
+ builder.setShuffleCheckRatio(shuffleCheckRatio)
+
+ builder.build()
+ }
+
+ @throws[IOException]
+ def combinedWrite(wholeStageIteratorWrapper: WholeStageIteratorWrapper[Product2[K, V]]): Unit = {
+ val itrHandle = wholeStageIteratorWrapper.inner.itrHandle()
+ shuffleWriterJniWrapper.addShuffleWriter(
+ itrHandle,
+ getShuffleWriterInfo().toByteArray,
+ celebornPartitionPusher)
+ if (wholeStageIteratorWrapper.hasNext) {
+ wholeStageIteratorWrapper.next()
+ assert(wholeStageIteratorWrapper.hasNext)
+ }
+ val result =
+ ShuffleWriterResult.parseFrom(shuffleWriterJniWrapper.getShuffleWriterResult)
+ val metrics = result.getMetrics
+ if (metrics.getInputRowNumber == 0) {
+ handleEmptyIterator()
+ return
+ }
+ writeMetrics.incRecordsWritten(metrics.getInputRowNumber)
+ writeMetrics.incWriteTime(metrics.getTotalWriteTime + metrics.getTotalPushTime)
+ dep.metrics("numInputRows").add(metrics.getInputRowNumber)
+ dep.metrics("dataSize").add(metrics.getDataSize)
+ dep.metrics("compressTime").add(metrics.getCompressTime)
+ dep.metrics("rssWriteTime").add(metrics.getTotalWriteTime)
+ dep.metrics("rssPushTime").add(metrics.getTotalPushTime)
+ partitionLengths = result.getPartitionLengthsList.asScala.toArray.map(l => l.toLong)
+ val startNs = System.nanoTime
+ pushMergedDataToCeleborn()
+ dep.metrics("rssCloseWaitTime").add(System.nanoTime() - startNs)
+ mapStatus = MapStatus(blockManager.shuffleServerId, partitionLengths, mapId)
+ }
+
+ @throws[IOException]
+ override def internalWrite(records: Iterator[Product2[K, V]]): Unit = {
+ records match {
+ case wrapper: WholeStageIteratorWrapper[Product2[K, V]] =>
+ // offload writer into WholeStageIterator and run as a Bolt operator
+ combinedWrite(wrapper)
+ return
+ case _ => ()
+ }
+ if (!records.hasNext) {
+ handleEmptyIterator()
+ return
+ }
+ while (records.hasNext) {
+ val cb = records.next()._2.asInstanceOf[ColumnarBatch]
+ if (cb.numRows == 0 || cb.numCols == 0) {
+ logInfo(s"Skip ColumnarBatch of ${cb.numRows} rows, ${cb.numCols} cols")
+ } else {
+ val columnarBatchHandle =
+ ColumnarBatches.getNativeHandle(BackendsApiManager.getBackendName, cb)
+ if (nativeShuffleWriter == -1L) {
+ createShuffleWriter(columnarBatchHandle)
+ }
+ val startTime = System.nanoTime()
+ shuffleWriterJniWrapper.write(
+ nativeShuffleWriter,
+ cb.numRows,
+ columnarBatchHandle,
+ availableOffHeapPerTask())
+ dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
+ dep.metrics("numInputRows").add(cb.numRows)
+ dep.metrics("inputBatches").add(1)
+ // This metric is important, AQE use it to decide if EliminateLimit
+ writeMetrics.incRecordsWritten(cb.numRows())
+ }
+ }
+
+ // If all of the ColumnarBatch have empty rows, the nativeShuffleWriter still equals -1
+ if (nativeShuffleWriter == -1L) {
+ handleEmptyIterator()
+ return
+ }
+
+ val startTime = System.nanoTime()
+ splitResult = shuffleWriterJniWrapper.stop(nativeShuffleWriter)
+
+ dep.metrics("shuffleWallTime").add(System.nanoTime() - startTime)
+ dep
+ .metrics("splitTime")
+ .add(
+ dep.metrics("shuffleWallTime").value - splitResult.getTotalPushTime -
+ splitResult.getTotalWriteTime -
+ splitResult.getTotalCompressTime)
+ dep.metrics("dataSize").add(splitResult.getRawPartitionLengths.sum)
+ writeMetrics.incBytesWritten(splitResult.getTotalBytesWritten)
+ writeMetrics.incWriteTime(splitResult.getTotalWriteTime + splitResult.getTotalPushTime)
+
+ partitionLengths = splitResult.getPartitionLengths
+
+ pushMergedDataToCeleborn()
+ mapStatus = MapStatus(blockManager.shuffleServerId, partitionLengths, mapId)
+
+ }
+
+ def createShuffleWriter(columnarBatchHandler: Long): Unit = {
+ nativeShuffleWriter = shuffleWriterJniWrapper.createShuffleWriter(
+ getShuffleWriterInfo().toByteArray,
+ columnarBatchHandler,
+ celebornPartitionPusher
+ )
+ runtime
+ .memoryManager()
+ .addSpiller(new Spiller() {
+ override def spill(self: MemoryTarget, phase: Spiller.Phase, size: Long): Long =
+ phase match {
+ case Spiller.Phase.SPILL =>
+ logInfo(s"Gluten shuffle writer: Trying to spill $size bytes of data")
+ val spilled = shuffleWriterJniWrapper.reclaim(nativeShuffleWriter, size)
+ logInfo(s"Gluten shuffle writer: Spilled $spilled / $size bytes of data")
+ spilled
+ case _ => 0L
+ }
+ })
+ }
+
+ override def closeShuffleWriter(): Unit = {
+ shuffleWriterJniWrapper.close(nativeShuffleWriter)
+ }
+}
diff --git a/gluten-ut/common/src/test/scala/org/apache/gluten/utils/BackendTestUtils.scala b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarShuffleWriterFactory.scala
similarity index 51%
copy from gluten-ut/common/src/test/scala/org/apache/gluten/utils/BackendTestUtils.scala
copy to backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarShuffleWriterFactory.scala
index 7635316610..926e32e770 100644
--- a/gluten-ut/common/src/test/scala/org/apache/gluten/utils/BackendTestUtils.scala
+++ b/backends-bolt/src-celeborn/main/scala/org/apache/spark/shuffle/BoltCelebornColumnarShuffleWriterFactory.scala
@@ -14,20 +14,29 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.gluten.utils
+package org.apache.spark.shuffle
-import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.spark.TaskContext
+import org.apache.spark.shuffle.celeborn.CelebornShuffleHandle
-// FIXME: We mixed the use of "Utils" and "Util" in utility class naming
-// FIXME: The utility methods "is...BackendLoaded" stands because we rely
-// on legacy backend-specific code in UTs. The new usages of the methods should
-// be better avoided. We'd remove them until the time is ripe.
-object BackendTestUtils {
- def isVeloxBackendLoaded(): Boolean = {
- BackendsApiManager.getBackendName == "velox" // the same with VeloxBackend.BACKEND_NAME
- }
+import org.apache.celeborn.client.ShuffleClient
+import org.apache.celeborn.common.CelebornConf
+
+class BoltCelebornColumnarShuffleWriterFactory extends SharedCelebornColumnarShuffleWriterFactory {
- def isCHBackendLoaded(): Boolean = {
- BackendsApiManager.getBackendName == "ch" // the same with CHBackend.BACKEND_NAME
+ override protected def createBackendShuffleWriter[K, V](
+ shuffleId: Int,
+ handle: CelebornShuffleHandle[K, V, V],
+ context: TaskContext,
+ celebornConf: CelebornConf,
+ client: ShuffleClient,
+ writeMetrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = {
+ new BoltCelebornColumnarShuffleWriter[K, V](
+ shuffleId,
+ handle,
+ context,
+ celebornConf,
+ client,
+ writeMetrics)
}
}
diff --git a/backends-bolt/src-delta/main/resources/META-INF/gluten-components/org.apache.gluten.component.BoltDeltaComponent b/backends-bolt/src-delta/main/resources/META-INF/gluten-components/org.apache.gluten.component.BoltDeltaComponent
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/backends-bolt/src-delta/main/scala/org/apache/gluten/component/BoltDeltaComponent.scala b/backends-bolt/src-delta/main/scala/org/apache/gluten/component/BoltDeltaComponent.scala
new file mode 100644
index 0000000000..fc89d66a29
--- /dev/null
+++ b/backends-bolt/src-delta/main/scala/org/apache/gluten/component/BoltDeltaComponent.scala
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.component
+
+import org.apache.gluten.backendsapi.bolt.BoltBackend
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.extension.{DeltaPostTransformRules, OffloadDeltaFilter, OffloadDeltaProject, OffloadDeltaScan}
+import org.apache.gluten.extension.columnar.enumerated.RasOffload
+import org.apache.gluten.extension.columnar.heuristic.HeuristicTransform
+import org.apache.gluten.extension.columnar.validator.Validators
+import org.apache.gluten.extension.injector.Injector
+
+import org.apache.spark.sql.execution.{FileSourceScanExec, FilterExec, ProjectExec}
+
+class BoltDeltaComponent extends Component {
+ override def name(): String = "bolt-delta"
+ override def buildInfo(): Component.BuildInfo =
+ Component.BuildInfo("BoltDelta", "N/A", "N/A", "N/A")
+ override def dependencies(): Seq[Class[_ <: Component]] = classOf[BoltBackend] :: Nil
+ override def injectRules(injector: Injector): Unit = {
+ val legacy = injector.gluten.legacy
+ val ras = injector.gluten.ras
+ legacy.injectTransform {
+ c =>
+ val offload = Seq(OffloadDeltaScan(), OffloadDeltaProject(), OffloadDeltaFilter())
+ .map(_.toStrcitRule())
+ HeuristicTransform.Simple(
+ Validators.newValidator(new GlutenConfig(c.sqlConf), offload),
+ offload)
+ }
+ val offloads: Seq[RasOffload] = Seq(
+ RasOffload.from[FileSourceScanExec](OffloadDeltaScan()),
+ RasOffload.from[ProjectExec](OffloadDeltaProject()),
+ RasOffload.from[FilterExec](OffloadDeltaFilter())
+ )
+ offloads.foreach(
+ offload =>
+ ras.injectRasRule(
+ c => RasOffload.Rule(offload, Validators.newValidator(new GlutenConfig(c.sqlConf)), Nil)))
+ DeltaPostTransformRules.rules.foreach {
+ r =>
+ legacy.injectPostTransform(_ => r)
+ ras.injectPostTransform(_ => r)
+ }
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-delta/test/scala/org/apache/gluten/execution/BoltDeltaSuite.scala
similarity index 81%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-delta/test/scala/org/apache/gluten/execution/BoltDeltaSuite.scala
index 632c806075..12af28bd79 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-delta/test/scala/org/apache/gluten/execution/BoltDeltaSuite.scala
@@ -14,13 +14,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.execution
-#pragma once
-
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+class BoltDeltaSuite extends DeltaSuite
diff --git a/backends-bolt/src-delta/test/scala/org/apache/gluten/execution/BoltTPCHDeltaSuite.scala b/backends-bolt/src-delta/test/scala/org/apache/gluten/execution/BoltTPCHDeltaSuite.scala
new file mode 100644
index 0000000000..3d9feae51d
--- /dev/null
+++ b/backends-bolt/src-delta/test/scala/org/apache/gluten/execution/BoltTPCHDeltaSuite.scala
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.spark.SparkConf
+
+import java.io.File
+
+class BoltTPCHDeltaSuite extends BoltTPCHSuite {
+ protected val tpchBasePath: String =
+ getClass.getResource("/").getPath + "../../../src/test/resources"
+
+ override protected val resourcePath: String =
+ getClass.getResource("/tpch-data-parquet").getFile
+
+ override protected val queriesResults: String =
+ new File(tpchBasePath, "queries-output").getCanonicalPath
+
+ override protected def sparkConf: SparkConf = {
+ super.sparkConf
+ .set("spark.executor.memory", "4g")
+ .set("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
+ .set("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
+ }
+
+ override protected def createTPCHNotNullTables(): Unit = {
+ val tableDir = getClass.getResource(resourcePath).getFile
+ TPCHTables
+ .map(_.name)
+ .map {
+ table =>
+ val tablePath = new File(tableDir, table).getAbsolutePath
+ val tableDF = spark.read.format(fileFormat).load(tablePath)
+ tableDF.write.format("delta").mode("append").saveAsTable(table)
+ (table, tableDF)
+ }
+ .toMap
+ }
+
+ override protected def afterAll(): Unit = {
+ TPCHTables.map(_.name).foreach(table => spark.sql(s"DROP TABLE IF EXISTS $table"))
+ super.afterAll()
+ }
+}
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeleteSQLSuite.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeleteSQLSuite.scala
new file mode 100644
index 0000000000..7bd6a59b7b
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeleteSQLSuite.scala
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.{DeltaExcludedTestMixin, DeltaSQLCommandTest}
+
+// spotless:off
+class DeleteSQLSuite extends DeleteSuiteBase
+ with DeltaExcludedTestMixin
+ with DeltaSQLCommandTest {
+
+ import testImplicits._
+
+ override protected def executeDelete(target: String, where: String = null): Unit = {
+ val whereClause = Option(where).map(c => s"WHERE $c").getOrElse("")
+ sql(s"DELETE FROM $target $whereClause")
+ }
+
+ override def excluded: Seq[String] = super.excluded ++
+ Seq(
+ // FIXME: Excluded by Gluten as results are mismatch.
+ "test delete on temp view - nontrivial projection - SQL TempView",
+ "test delete on temp view - nontrivial projection - Dataset TempView"
+ )
+
+ // For EXPLAIN, which is not supported in OSS
+ test("explain") {
+ append(Seq((2, 2)).toDF("key", "value"))
+ val df = sql(s"EXPLAIN DELETE FROM delta.`$tempPath` WHERE key = 2")
+ val outputs = df.collect().map(_.mkString).mkString
+ assert(outputs.contains("Delta"))
+ assert(!outputs.contains("index") && !outputs.contains("ActionLog"))
+ // no change should be made by explain
+ checkAnswer(readDeltaTable(tempPath), Row(2, 2))
+ }
+
+ test("delete from a temp view") {
+ withTable("tab") {
+ withTempView("v") {
+ Seq((1, 1), (0, 3), (1, 5)).toDF("key", "value").write.format("delta").saveAsTable("tab")
+ spark.table("tab").as("name").createTempView("v")
+ sql("DELETE FROM v WHERE key = 1")
+ checkAnswer(spark.table("tab"), Row(0, 3))
+ }
+ }
+ }
+
+ test("delete from a SQL temp view") {
+ withTable("tab") {
+ withTempView("v") {
+ Seq((1, 1), (0, 3), (1, 5)).toDF("key", "value").write.format("delta").saveAsTable("tab")
+ sql("CREATE TEMP VIEW v AS SELECT * FROM tab")
+ sql("DELETE FROM v WHERE key = 1 AND VALUE = 5")
+ checkAnswer(spark.table("tab"), Seq(Row(1, 1), Row(0, 3)))
+ }
+ }
+ }
+
+ Seq(true, false).foreach { partitioned =>
+ test(s"User defined _change_type column doesn't get dropped - partitioned=$partitioned") {
+ withTable("tab") {
+ sql(
+ s"""CREATE TABLE tab USING DELTA
+ |${if (partitioned) "PARTITIONED BY (part) " else ""}
+ |TBLPROPERTIES (delta.enableChangeDataFeed = false)
+ |AS SELECT id, int(id / 10) AS part, 'foo' as _change_type
+ |FROM RANGE(1000)
+ |""".stripMargin)
+ val rowsToDelete = (1 to 1000 by 42).mkString("(", ", ", ")")
+ executeDelete("tab", s"id in $rowsToDelete")
+ sql("SELECT id, _change_type FROM tab").collect().foreach { row =>
+ val _change_type = row.getString(1)
+ assert(_change_type === "foo", s"Invalid _change_type for id=${row.get(0)}")
+ }
+ }
+ }
+ }
+}
+
+class DeleteSQLNameColumnMappingSuite extends DeleteSQLSuite
+ with DeltaColumnMappingEnableNameMode {
+
+ protected override def runOnlyTests: Seq[String] = Seq(true, false).map { isPartitioned =>
+ s"basic case - delete from a Delta table by name - Partition=$isPartitioned"
+ } ++ Seq(true, false).flatMap { isPartitioned =>
+ Seq(
+ s"where key columns - Partition=$isPartitioned",
+ s"where data columns - Partition=$isPartitioned")
+ }
+
+}
+
+class DeleteSQLWithDeletionVectorsSuite extends DeleteSQLSuite
+ with DeltaExcludedTestMixin
+ with DeletionVectorsTestUtils {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ enableDeletionVectors(spark, delete = true)
+ spark.conf.set(DeltaSQLConf.DELETION_VECTORS_USE_METADATA_ROW_INDEX.key, "false")
+ }
+
+ override def excluded: Seq[String] = super.excluded ++
+ Seq(
+ // The following two tests must fail when DV is used. Covered by another test case:
+ // "throw error when non-pinned TahoeFileIndex snapshot is used".
+ "data and partition columns - Partition=true Skipping=false",
+ "data and partition columns - Partition=false Skipping=false",
+ // The scan schema contains additional row index filter columns.
+ "nested schema pruning on data condition",
+ // The number of records is not recomputed when using DVs
+ "delete throws error if number of records increases",
+ "delete logs error if number of records are missing in stats",
+ // FIXME: Excluded by Gluten as results are mismatch.
+ "test delete on temp view - nontrivial projection - SQL TempView",
+ "test delete on temp view - nontrivial projection - Dataset TempView"
+ )
+
+ // This works correctly with DVs, but fails in classic DELETE.
+ override def testSuperSetColsTempView(): Unit = {
+ testComplexTempViews("superset cols")(
+ text = "SELECT key, value, 1 FROM tab",
+ expectResult = Row(0, 3, 1) :: Nil)
+ }
+}
+
+class DeleteSQLWithDeletionVectorsAndPredicatePushdownSuite
+ extends DeleteSQLWithDeletionVectorsSuite {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set(DeltaSQLConf.DELETION_VECTORS_USE_METADATA_ROW_INDEX.key, "true")
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeleteSuiteBase.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeleteSuiteBase.scala
new file mode 100644
index 0000000000..8ab9510ff3
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeleteSuiteBase.scala
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.{AnalysisException, DataFrame, QueryTest, Row}
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.execution.FileSourceScanExec
+import org.apache.spark.sql.functions.{lit, struct}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.StructType
+
+import shims.DeltaExcludedBySparkVersionTestMixinShims
+
+// spotless:off
+abstract class DeleteSuiteBase extends QueryTest
+ with SharedSparkSession
+ with DeltaDMLTestUtils
+ with DeltaTestUtilsForTempViews
+ with DeltaExcludedBySparkVersionTestMixinShims {
+
+ import testImplicits._
+
+ protected def executeDelete(target: String, where: String = null): Unit
+
+ protected def checkDelete(
+ condition: Option[String],
+ expectedResults: Seq[Row],
+ tableName: Option[String] = None): Unit = {
+ executeDelete(target = tableName.getOrElse(s"delta.`$tempPath`"), where = condition.orNull)
+ checkAnswer(readDeltaTable(tempPath), expectedResults)
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"basic case - Partition=$isPartitioned") {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"), partitions)
+
+ checkDelete(condition = None, Nil)
+ }
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"basic case - delete from a Delta table by path - Partition=$isPartitioned") {
+ withTable("deltaTable") {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ append(input, partitions)
+
+ checkDelete(Some("value = 4 and key = 3"),
+ Row(2, 2) :: Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 4 and key = 1"),
+ Row(2, 2) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 2 or key = 1"),
+ Row(0, 3) :: Nil)
+ checkDelete(Some("key = 0 or value = 99"), Nil)
+ }
+ }
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"basic case - delete from a Delta table by name - Partition=$isPartitioned") {
+ withTable("delta_table") {
+ val partitionByClause = if (isPartitioned) "PARTITIONED BY (key)" else ""
+ sql(
+ s"""
+ |CREATE TABLE delta_table(key INT, value INT)
+ |USING delta
+ |OPTIONS('path'='$tempPath')
+ |$partitionByClause
+ """.stripMargin)
+
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ append(input)
+
+ checkDelete(Some("value = 4 and key = 3"),
+ Row(2, 2) :: Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil,
+ Some("delta_table"))
+ checkDelete(Some("value = 4 and key = 1"),
+ Row(2, 2) :: Row(1, 1) :: Row(0, 3) :: Nil,
+ Some("delta_table"))
+ checkDelete(Some("value = 2 or key = 1"),
+ Row(0, 3) :: Nil,
+ Some("delta_table"))
+ checkDelete(Some("key = 0 or value = 99"),
+ Nil,
+ Some("delta_table"))
+ }
+ }
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"basic key columns - Partition=$isPartitioned") {
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(input, partitions)
+
+ checkDelete(Some("key > 2"), Row(2, 2) :: Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("key < 2"), Row(2, 2) :: Nil)
+ checkDelete(Some("key = 2"), Nil)
+ }
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"where key columns - Partition=$isPartitioned") {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"), partitions)
+
+ checkDelete(Some("key = 1"), Row(2, 2) :: Row(0, 3) :: Nil)
+ checkDelete(Some("key = 2"), Row(0, 3) :: Nil)
+ checkDelete(Some("key = 0"), Nil)
+ }
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"where data columns - Partition=$isPartitioned") {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"), partitions)
+
+ checkDelete(Some("value <= 2"), Row(1, 4) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 3"), Row(1, 4) :: Nil)
+ checkDelete(Some("value != 0"), Nil)
+ }
+ }
+
+ test("where data columns and partition columns") {
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ append(input, Seq("key"))
+
+ checkDelete(Some("value = 4 and key = 3"),
+ Row(2, 2) :: Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 4 and key = 1"),
+ Row(2, 2) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 2 or key = 1"),
+ Row(0, 3) :: Nil)
+ checkDelete(Some("key = 0 or value = 99"),
+ Nil)
+ }
+
+ Seq(true, false).foreach { skippingEnabled =>
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"data and partition columns - Partition=$isPartitioned Skipping=$skippingEnabled") {
+ withSQLConf(DeltaSQLConf.DELTA_STATS_SKIPPING.key -> skippingEnabled.toString) {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ append(input, partitions)
+
+ checkDelete(Some("value = 4 and key = 3"),
+ Row(2, 2) :: Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 4 and key = 1"),
+ Row(2, 2) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ checkDelete(Some("value = 2 or key = 1"),
+ Row(0, 3) :: Nil)
+ checkDelete(Some("key = 0 or value = 99"),
+ Nil)
+ }
+ }
+ }
+ }
+
+ test("Negative case - non-Delta target") {
+ Seq((1, 1), (0, 3), (1, 5)).toDF("key1", "value")
+ .write.format("parquet").mode("append").save(tempPath)
+ val e = intercept[DeltaAnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`")
+ }.getMessage
+ assert(e.contains("DELETE destination only supports Delta sources") ||
+ e.contains("is not a Delta table") || e.contains("doesn't exist") ||
+ e.contains("Incompatible format"))
+ }
+
+ test("Negative case - non-deterministic condition") {
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"))
+ val e = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`", where = "rand() > 0.5")
+ }.getMessage
+ assert(e.contains("nondeterministic expressions are only allowed in") ||
+ e.contains("The operator expects a deterministic expression"))
+ }
+
+ test("Negative case - DELETE the child directory") {
+ append(Seq((2, 2), (3, 2)).toDF("key", "value"), partitionBy = "key" :: Nil)
+ val e = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath/key=2`", where = "value = 2")
+ }.getMessage
+ assert(e.contains("Expect a full scan of Delta sources, but found a partial scan"))
+ }
+
+ test("delete cached table by name") {
+ withTable("cached_delta_table") {
+ Seq((2, 2), (1, 4)).toDF("key", "value")
+ .write.format("delta").saveAsTable("cached_delta_table")
+
+ spark.table("cached_delta_table").cache()
+ spark.table("cached_delta_table").collect()
+ executeDelete(target = "cached_delta_table", where = "key = 2")
+ checkAnswer(spark.table("cached_delta_table"), Row(1, 4) :: Nil)
+ }
+ }
+
+ test("delete cached table by path") {
+ Seq((2, 2), (1, 4)).toDF("key", "value")
+ .write.mode("overwrite").format("delta").save(tempPath)
+ spark.read.format("delta").load(tempPath).cache()
+ spark.read.format("delta").load(tempPath).collect()
+ executeDelete(s"delta.`$tempPath`", where = "key = 2")
+ checkAnswer(spark.read.format("delta").load(tempPath), Row(1, 4) :: Nil)
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"condition having current_date - Partition=$isPartitioned") {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(
+ Seq((java.sql.Date.valueOf("1969-12-31"), 2),
+ (java.sql.Date.valueOf("2099-12-31"), 4))
+ .toDF("key", "value"), partitions)
+
+ checkDelete(Some("CURRENT_DATE > key"),
+ Row(java.sql.Date.valueOf("2099-12-31"), 4) :: Nil)
+ checkDelete(Some("CURRENT_DATE <= key"), Nil)
+ }
+ }
+
+ test("condition having current_timestamp - Partition by Timestamp") {
+ append(
+ Seq((java.sql.Timestamp.valueOf("2012-12-31 16:00:10.011"), 2),
+ (java.sql.Timestamp.valueOf("2099-12-31 16:00:10.011"), 4))
+ .toDF("key", "value"), Seq("key"))
+
+ checkDelete(Some("CURRENT_TIMESTAMP > key"),
+ Row(java.sql.Timestamp.valueOf("2099-12-31 16:00:10.011"), 4) :: Nil)
+ checkDelete(Some("CURRENT_TIMESTAMP <= key"), Nil)
+ }
+
+ Seq(true, false).foreach { isPartitioned =>
+ test(s"foldable condition - Partition=$isPartitioned") {
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"), partitions)
+
+ val allRows = Row(2, 2) :: Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil
+
+ checkDelete(Some("false"), allRows)
+ checkDelete(Some("1 <> 1"), allRows)
+ checkDelete(Some("1 > null"), allRows)
+ checkDelete(Some("true"), Nil)
+ checkDelete(Some("1 = 1"), Nil)
+ }
+ }
+
+ test("SC-12232: should not delete the rows where condition evaluates to null") {
+ append(Seq(("a", null), ("b", null), ("c", "v"), ("d", "vv")).toDF("key", "value").coalesce(1))
+
+ // "null = null" evaluates to null
+ checkDelete(Some("value = null"),
+ Row("a", null) :: Row("b", null) :: Row("c", "v") :: Row("d", "vv") :: Nil)
+
+ // these expressions evaluate to null when value is null
+ checkDelete(Some("value = 'v'"),
+ Row("a", null) :: Row("b", null) :: Row("d", "vv") :: Nil)
+ checkDelete(Some("value <> 'v'"),
+ Row("a", null) :: Row("b", null) :: Nil)
+ }
+
+ test("SC-12232: delete rows with null values using isNull") {
+ append(Seq(("a", null), ("b", null), ("c", "v"), ("d", "vv")).toDF("key", "value").coalesce(1))
+
+ // when value is null, this expression evaluates to true
+ checkDelete(Some("value is null"),
+ Row("c", "v") :: Row("d", "vv") :: Nil)
+ }
+
+ test("SC-12232: delete rows with null values using EqualNullSafe") {
+ append(Seq(("a", null), ("b", null), ("c", "v"), ("d", "vv")).toDF("key", "value").coalesce(1))
+
+ // when value is null, this expression evaluates to true
+ checkDelete(Some("value <=> null"),
+ Row("c", "v") :: Row("d", "vv") :: Nil)
+ }
+
+ test("do not support subquery test") {
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"))
+ Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("c", "d").createOrReplaceTempView("source")
+
+ // basic subquery
+ val e0 = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`", "key < (SELECT max(c) FROM source)")
+ }.getMessage
+ assert(e0.contains("Subqueries are not supported"))
+
+ // subquery with EXISTS
+ val e1 = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`", "EXISTS (SELECT max(c) FROM source)")
+ }.getMessage
+ assert(e1.contains("Subqueries are not supported"))
+
+ // subquery with NOT EXISTS
+ val e2 = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`", "NOT EXISTS (SELECT max(c) FROM source)")
+ }.getMessage
+ assert(e2.contains("Subqueries are not supported"))
+
+ // subquery with IN
+ val e3 = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`", "key IN (SELECT max(c) FROM source)")
+ }.getMessage
+ assert(e3.contains("Subqueries are not supported"))
+
+ // subquery with NOT IN
+ val e4 = intercept[AnalysisException] {
+ executeDelete(target = s"delta.`$tempPath`", "key NOT IN (SELECT max(c) FROM source)")
+ }.getMessage
+ assert(e4.contains("Subqueries are not supported"))
+ }
+
+ test("schema pruning on data condition") {
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ append(input, Nil)
+ // Start from a cached snapshot state
+ deltaLog.update().stateDF
+
+ val executedPlans = DeltaTestUtils.withPhysicalPlansCaptured(spark) {
+ checkDelete(Some("key = 2"),
+ Row(1, 4) :: Row(1, 1) :: Row(0, 3) :: Nil)
+ }
+
+ val scans = executedPlans.flatMap(_.collect {
+ case f: FileSourceScanExec => f
+ })
+
+ // The first scan is for finding files to delete. We only are matching against the key
+ // so that should be the only field in the schema
+ assert(scans.head.schema.findNestedField(Seq("key")).nonEmpty)
+ assert(scans.head.schema.findNestedField(Seq("value")).isEmpty)
+ }
+
+
+ test("nested schema pruning on data condition") {
+ val input = Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value")
+ .select(struct("key", "value").alias("nested"))
+ append(input, Nil)
+ // Start from a cached snapshot state
+ deltaLog.update().stateDF
+
+ val executedPlans = DeltaTestUtils.withPhysicalPlansCaptured(spark) {
+ checkDelete(Some("nested.key = 2"),
+ Row(Row(1, 4)) :: Row(Row(1, 1)) :: Row(Row(0, 3)) :: Nil)
+ }
+
+ val scans = executedPlans.flatMap(_.collect {
+ case f: FileSourceScanExec => f
+ })
+
+ assert(scans.head.schema == StructType.fromDDL("nested STRUCT<key: int>"))
+ }
+
+ /**
+ * @param function the unsupported function.
+ * @param functionType The type of the unsupported expression to be tested.
+ * @param data the data in the table.
+ * @param where the where clause containing the unsupported expression.
+ * @param expectException whether an exception is expected to be thrown
+ * @param customErrorRegex customized error regex.
+ */
+ def testUnsupportedExpression(
+ function: String,
+ functionType: String,
+ data: => DataFrame,
+ where: String,
+ expectException: Boolean,
+ customErrorRegex: Option[String] = None): Unit = {
+ test(s"$functionType functions in delete - expect exception: $expectException") {
+ withTable("deltaTable") {
+ data.write.format("delta").saveAsTable("deltaTable")
+
+ val expectedErrorRegex = "(?s).*(?i)unsupported.*(?i).*Invalid expressions.*"
+
+ var catchException = true
+
+ var errorRegex = if (functionType.equals("Generate")) {
+ ".*Subqueries are not supported in the DELETE.*"
+ } else customErrorRegex.getOrElse(expectedErrorRegex)
+
+
+ if (catchException) {
+ val dataBeforeException = spark.read.format("delta").table("deltaTable").collect()
+ val e = intercept[Exception] {
+ executeDelete(target = "deltaTable", where = where)
+ }
+ val message = if (e.getCause != null) {
+ e.getCause.getMessage
+ } else e.getMessage
+ assert(message.matches(errorRegex))
+ checkAnswer(spark.read.format("delta").table("deltaTable"), dataBeforeException)
+ } else {
+ executeDelete(target = "deltaTable", where = where)
+ }
+ }
+ }
+ }
+
+ testUnsupportedExpression(
+ function = "row_number",
+ functionType = "Window",
+ data = Seq((2, 2), (1, 4)).toDF("key", "value"),
+ where = "row_number() over (order by value) > 1",
+ expectException = true
+ )
+
+ testUnsupportedExpression(
+ function = "max",
+ functionType = "Aggregate",
+ data = Seq((2, 2), (1, 4)).toDF("key", "value"),
+ where = "key > max(value)",
+ expectException = true
+ )
+
+ // Explode functions are supported in where if only one row generated.
+ testUnsupportedExpression(
+ function = "explode",
+ functionType = "Generate",
+ data = Seq((2, List(2))).toDF("key", "value"),
+ where = "key = (select explode(value) from deltaTable)",
+ expectException = false // generate only one row, no exception.
+ )
+
+ // Explode functions are supported in where but if there's more than one row generated,
+ // it will throw an exception.
+ testUnsupportedExpression(
+ function = "explode",
+ functionType = "Generate",
+ data = Seq((2, List(2)), (1, List(4, 5))).toDF("key", "value"),
+ where = "key = (select explode(value) from deltaTable)",
+ expectException = true, // generate more than one row. Exception expected.
+ customErrorRegex =
+ Some(".*More than one row returned by a subquery used as an expression(?s).*")
+ )
+
+ Seq(true, false).foreach { isPartitioned =>
+ val name = s"test delete on temp view - basic - Partition=$isPartitioned"
+ testWithTempView(name) { isSQLTempView =>
+ val partitions = if (isPartitioned) "key" :: Nil else Nil
+ append(Seq((2, 2), (1, 4), (1, 1), (0, 3)).toDF("key", "value"), partitions)
+ createTempViewFromTable(s"delta.`$tempPath`", isSQLTempView)
+ checkDelete(
+ condition = Some("key <= 1"),
+ expectedResults = Row(2, 2) :: Nil,
+ tableName = Some("v"))
+ }
+ }
+
+ protected def testInvalidTempViews(name: String)(
+ text: String,
+ expectedErrorMsgForSQLTempView: String = null,
+ expectedErrorMsgForDataSetTempView: String = null,
+ expectedErrorClassForSQLTempView: String = null,
+ expectedErrorClassForDataSetTempView: String = null): Unit = {
+ testWithTempView(s"test delete on temp view - $name") { isSQLTempView =>
+ withTable("tab") {
+ Seq((0, 3), (1, 2)).toDF("key", "value").write.format("delta").saveAsTable("tab")
+ if (isSQLTempView) {
+ sql(s"CREATE TEMP VIEW v AS $text")
+ } else {
+ sql(text).createOrReplaceTempView("v")
+ }
+ val ex = intercept[AnalysisException] {
+ executeDelete(
+ "v",
+ "key >= 1 and value < 3"
+ )
+ }
+ testErrorMessageAndClass(
+ isSQLTempView,
+ ex,
+ expectedErrorMsgForSQLTempView,
+ expectedErrorMsgForDataSetTempView,
+ expectedErrorClassForSQLTempView,
+ expectedErrorClassForDataSetTempView)
+ }
+ }
+ }
+ testInvalidTempViews("subset cols")(
+ text = "SELECT key FROM tab",
+ expectedErrorClassForSQLTempView = "UNRESOLVED_COLUMN.WITH_SUGGESTION",
+ expectedErrorClassForDataSetTempView = "UNRESOLVED_COLUMN.WITH_SUGGESTION"
+ )
+
+ // Need to be able to override this, because it works in some configurations.
+ protected def testSuperSetColsTempView(): Unit = {
+ testInvalidTempViews("superset cols")(
+ text = "SELECT key, value, 1 FROM tab",
+ // The analyzer can't tell whether the table originally had the extra column or not.
+ expectedErrorMsgForSQLTempView = "Can't resolve column 1 in root",
+ expectedErrorMsgForDataSetTempView = "Can't resolve column 1 in root"
+ )
+ }
+
+ testSuperSetColsTempView()
+
+ protected def testComplexTempViews(name: String)(
+ text: String,
+ expectResult: Seq[Row]): Unit = {
+ testWithTempView(s"test delete on temp view - $name") { isSQLTempView =>
+ withTable("tab") {
+ Seq((0, 3), (1, 2)).toDF("key", "value").write.format("delta").saveAsTable("tab")
+ createTempViewFromSelect(text, isSQLTempView)
+ executeDelete(
+ "v",
+ "key >= 1 and value < 3"
+ )
+ checkAnswer(spark.read.format("delta").table("v"), expectResult)
+ }
+ }
+ }
+
+ testComplexTempViews("nontrivial projection")(
+ text = "SELECT value as key, key as value FROM tab",
+ expectResult = Row(3, 0) :: Nil
+ )
+
+ testComplexTempViews("view with too many internal aliases")(
+ text = "SELECT * FROM (SELECT * FROM tab AS t1) AS t2",
+ expectResult = Row(0, 3) :: Nil
+ )
+
+ testSparkMasterOnly("Variant type") {
+ val dstDf = sql(
+ """SELECT parse_json(cast(id as string)) v, id i
+ FROM range(3)""")
+ append(dstDf)
+
+ executeDelete(target = s"delta.`$tempPath`", where = "to_json(v) = '1'")
+
+ checkAnswer(readDeltaTable(tempPath).selectExpr("i", "to_json(v)"),
+ Seq(Row(0, "0"), Row(2, "2")))
+ }
+
+ test("delete on partitioned table with special chars") {
+ val partValue = "part%one"
+ spark.range(0, 3, 1, 1).toDF("key").withColumn("value", lit(partValue))
+ .write.format("delta").partitionBy("value").save(tempPath)
+ checkDelete(
+ condition = Some(s"value = '$partValue' and key = 1"),
+ expectedResults = Row(0, partValue) :: Row(2, partValue) :: Nil)
+ checkDelete(
+ condition = Some(s"value = '$partValue' and key = 2"),
+ expectedResults = Row(0, partValue) :: Nil)
+ checkDelete(
+ condition = Some(s"value = '$partValue'"),
+ expectedResults = Nil)
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeletionVectorsTestUtils.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeletionVectorsTestUtils.scala
new file mode 100644
index 0000000000..5bb022c12d
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeletionVectorsTestUtils.scala
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.sql.{DataFrame, QueryTest, RuntimeConfig, SparkSession}
+import org.apache.spark.sql.delta.DeltaOperations.Truncate
+import org.apache.spark.sql.delta.actions.{Action, AddFile, DeletionVectorDescriptor, RemoveFile}
+import org.apache.spark.sql.delta.deletionvectors.{RoaringBitmapArray, RoaringBitmapArrayFormat}
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.storage.dv.DeletionVectorStore
+import org.apache.spark.sql.delta.test.DeltaSQLTestUtils
+import org.apache.spark.sql.delta.util.PathWithFileSystem
+import org.apache.spark.sql.functions.{col, lit}
+import org.apache.spark.sql.test.SharedSparkSession
+
+import org.apache.commons.io.FileUtils
+import org.apache.hadoop.fs.Path
+
+import java.io.File
+import java.util.UUID
+
+// spotless:off
+/** Collection of test utilities related with persistent Deletion Vectors. */
+trait DeletionVectorsTestUtils extends QueryTest with SharedSparkSession with DeltaSQLTestUtils {
+
+ def enableDeletionVectors(
+ spark: SparkSession,
+ delete: Boolean = false,
+ update: Boolean = false,
+ merge: Boolean = false): Unit = {
+ val global = delete || update || merge
+ spark.conf
+ .set(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.defaultTablePropertyKey, global.toString)
+ spark.conf.set(DeltaSQLConf.DELETE_USE_PERSISTENT_DELETION_VECTORS.key, delete.toString)
+ spark.conf.set(DeltaSQLConf.UPDATE_USE_PERSISTENT_DELETION_VECTORS.key, update.toString)
+ spark.conf.set(DeltaSQLConf.MERGE_USE_PERSISTENT_DELETION_VECTORS.key, merge.toString)
+ }
+
+ def enableDeletionVectorsForAllSupportedOperations(spark: SparkSession): Unit =
+ enableDeletionVectors(spark, delete = true, update = true)
+
+ def testWithDVs(testName: String, testTags: org.scalatest.Tag*)(thunk: => Unit): Unit = {
+ test(testName, testTags : _*) {
+ withDeletionVectorsEnabled() {
+ thunk
+ }
+ }
+ }
+
+ /** Run a thunk with Deletion Vectors enabled/disabled. */
+ def withDeletionVectorsEnabled(enabled: Boolean = true)(thunk: => Unit): Unit = {
+ val enabledStr = enabled.toString
+ withSQLConf(
+ DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.defaultTablePropertyKey -> enabledStr,
+ DeltaSQLConf.DELETE_USE_PERSISTENT_DELETION_VECTORS.key -> enabledStr,
+ DeltaSQLConf.UPDATE_USE_PERSISTENT_DELETION_VECTORS.key -> enabledStr,
+ DeltaSQLConf.MERGE_USE_PERSISTENT_DELETION_VECTORS.key -> enabledStr) {
+ thunk
+ }
+ }
+
+ /** Helper to run 'fn' with a temporary Delta table. */
+ def withTempDeltaTable(
+ dataDF: DataFrame,
+ partitionBy: Seq[String] = Seq.empty,
+ enableDVs: Boolean = true,
+ conf: Seq[(String, String)] = Nil)
+ (fn: (() => io.delta.tables.DeltaTable, DeltaLog) => Unit): Unit = {
+ withTempPath { path =>
+ val tablePath = new Path(path.getAbsolutePath)
+ withSQLConf(conf: _*) {
+ dataDF.write
+ .option(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.key, enableDVs.toString)
+ .partitionBy(partitionBy: _*)
+ .format("delta")
+ .save(tablePath.toString)
+ }
+ // DeltaTable hangs on to the DataFrame it is created with for the entire object lifetime.
+ // That means subsequent `targetTable.toDF` calls will return the same snapshot.
+ // The DV tests are generally written assuming `targetTable.toDF` would return a new snapshot.
+ // So create a function here instead of a n instance, so `targetTable().toDF`
+ // will actually provide a new snapshot.
+ val targetTable =
+ () => io.delta.tables.DeltaTable.forPath(tablePath.toString)
+ val targetLog = DeltaLog.forTable(spark, tablePath)
+ fn(targetTable, targetLog)
+ }
+ }
+
+ /** Create a temp path which contains special characters. */
+ override def withTempPath(f: File => Unit): Unit = {
+ super.withTempPath(prefix = "s p a r k %2a")(f)
+ }
+
+ /** Create a temp path which contains special characters. */
+ override protected def withTempDir(f: File => Unit): Unit = {
+ super.withTempDir(prefix = "s p a r k %2a")(f)
+ }
+
+ /** Helper that verifies whether a defined number of DVs exist */
+ def verifyDVsExist(targetLog: DeltaLog, filesWithDVsSize: Int): Unit = {
+ val filesWithDVs = getFilesWithDeletionVectors(targetLog)
+ assert(filesWithDVs.size === filesWithDVsSize)
+ assertDeletionVectorsExist(targetLog, filesWithDVs)
+ }
+
+ /** Returns all [[AddFile]] actions of a Delta table that contain Deletion Vectors. */
+ def getFilesWithDeletionVectors(log: DeltaLog): Seq[AddFile] =
+ log.update().allFiles.collect().filter(_.deletionVector != null).toSeq
+
+ /** Lists the Deletion Vectors files of a table. */
+ def listDeletionVectors(log: DeltaLog): Seq[File] = {
+ val dir = new File(log.dataPath.toUri.getPath)
+ dir.listFiles().filter(_.getName.startsWith(
+ DeletionVectorDescriptor.DELETION_VECTOR_FILE_NAME_CORE))
+ }
+
+ /** Helper to check that the Deletion Vectors of the provided file actions exist on disk. */
+ def assertDeletionVectorsExist(log: DeltaLog, filesWithDVs: Seq[AddFile]): Unit = {
+ val tablePath = new Path(log.dataPath.toUri.getPath)
+ for (file <- filesWithDVs) {
+ val dv = file.deletionVector
+ assert(dv != null)
+ assert(dv.isOnDisk && !dv.isInline)
+ assert(dv.offset.isDefined)
+
+ // Check that DV exists.
+ val dvPath = dv.absolutePath(tablePath)
+ assert(new File(dvPath.toString).exists(), s"DV not found $dvPath")
+
+ // Check that cardinality is correct.
+ val bitmap = newDVStore.read(dvPath, dv.offset.get, dv.sizeInBytes)
+ assert(dv.cardinality === bitmap.cardinality)
+ }
+ }
+
+ /** Enable persistent deletion vectors in new Delta tables. */
+ def enableDeletionVectorsInNewTables(conf: RuntimeConfig): Unit =
+ conf.set(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.defaultTablePropertyKey, "true")
+
+ /** Enable persistent Deletion Vectors in a Delta table. */
+ def enableDeletionVectorsInTable(tablePath: Path, enable: Boolean): Unit =
+ spark.sql(
+ s"""ALTER TABLE delta.`$tablePath`
+ |SET TBLPROPERTIES ('${DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.key}' = '$enable')
+ |""".stripMargin)
+
+ /** Enable persistent Deletion Vectors in a Delta table. */
+ def enableDeletionVectorsInTable(deltaLog: DeltaLog, enable: Boolean = true): Unit =
+ enableDeletionVectorsInTable(deltaLog.dataPath, enable)
+
+ /** Enable persistent deletion vectors in new tables and DELETE DML commands. */
+ def enableDeletionVectors(conf: RuntimeConfig): Unit = {
+ enableDeletionVectorsInNewTables(conf)
+ conf.set(DeltaSQLConf.DELETE_USE_PERSISTENT_DELETION_VECTORS.key, "true")
+ }
+
+ // ======== HELPER METHODS TO WRITE DVs ==========
+ /** Helper method to remove the specified rows in the given file using DVs */
+ protected def removeRowsFromFileUsingDV(
+ log: DeltaLog,
+ addFile: AddFile,
+ rowIds: Seq[Long]): Seq[Action] = {
+ val dv = RoaringBitmapArray(rowIds: _*)
+ writeFileWithDV(log, addFile, dv)
+ }
+
+ /** Utility method to remove a ratio of rows from the given file */
+ protected def deleteRows(
+ log: DeltaLog, file: AddFile, approxPhyRows: Long, ratioOfRowsToDelete: Double): Unit = {
+ val numRowsToDelete =
+ Math.ceil(ratioOfRowsToDelete * file.numPhysicalRecords.getOrElse(approxPhyRows)).toInt
+ removeRowsFromFile(log, file, Seq.range(0, numRowsToDelete))
+ }
+
+ /** Utility method to remove the given rows from the given file using DVs */
+ protected def removeRowsFromFile(
+ log: DeltaLog, addFile: AddFile, rowIndexesToRemove: Seq[Long]): Unit = {
+ val txn = log.startTransaction()
+ val actions = removeRowsFromFileUsingDV(log, addFile, rowIndexesToRemove)
+ txn.commit(actions, Truncate())
+ }
+
+ protected def getFileActionsInLastVersion(log: DeltaLog): (Seq[AddFile], Seq[RemoveFile]) = {
+ val version = log.update().version
+ val allFiles = log.getChanges(version).toSeq.head._2
+ val add = allFiles.collect { case a: AddFile => a }
+ val remove = allFiles.collect { case r: RemoveFile => r }
+ (add, remove)
+ }
+
+ protected def serializeRoaringBitmapArrayWithDefaultFormat(
+ dv: RoaringBitmapArray): Array[Byte] = {
+ val serializationFormat = RoaringBitmapArrayFormat.Portable
+ dv.serializeAsByteArray(serializationFormat)
+ }
+
+ /**
+ * Produce a new [[AddFile]] that will store `dv` in the log using default settings for choosing
+ * inline or on-disk storage.
+ *
+ * Also returns the corresponding [[RemoveFile]] action for `currentFile`.
+ *
+ * TODO: Always on-disk for now. Inline support comes later.
+ */
+ protected def writeFileWithDV(
+ log: DeltaLog,
+ currentFile: AddFile,
+ dv: RoaringBitmapArray): Seq[Action] = {
+ writeFileWithDVOnDisk(log, currentFile, dv)
+ }
+
+ /** Name of the partition column used by [[createTestDF()]]. */
+ val PARTITION_COL = "partitionColumn"
+
+ def createTestDF(
+ start: Long,
+ end: Long,
+ numFiles: Int,
+ partitionColumn: Option[Int] = None): DataFrame = {
+ val df = spark.range(start, end, 1, numFiles).withColumn("v", col("id"))
+ if (partitionColumn.isEmpty) {
+ df
+ } else {
+ df.withColumn(PARTITION_COL, lit(partitionColumn.get))
+ }
+ }
+
+ /**
+ * Produce a new [[AddFile]] that will reference the `dv` in the log while storing it on-disk.
+ *
+ * Also returns the corresponding [[RemoveFile]] action for `currentFile`.
+ */
+ protected def writeFileWithDVOnDisk(
+ log: DeltaLog,
+ currentFile: AddFile,
+ dv: RoaringBitmapArray): Seq[Action] = writeFilesWithDVsOnDisk(log, Seq((currentFile, dv)))
+
+ protected def withDVWriter[T](
+ log: DeltaLog,
+ dvFileID: UUID)(fn: DeletionVectorStore.Writer => T): T = {
+ val dvStore = newDVStore
+ // scalastyle:off deltahadoopconfiguration
+ val conf = spark.sessionState.newHadoopConf()
+ // scalastyle:on deltahadoopconfiguration
+ val tableWithFS = PathWithFileSystem.withConf(log.dataPath, conf)
+ val dvPath =
+ DeletionVectorStore.assembleDeletionVectorPathWithFileSystem(tableWithFS, dvFileID)
+ val writer = dvStore.createWriter(dvPath)
+ try {
+ fn(writer)
+ } finally {
+ writer.close()
+ }
+ }
+
+ /**
+ * Produce new [[AddFile]] actions that will reference associated DVs in the log while storing
+ * all DVs in the same file on-disk.
+ *
+ * Also returns the corresponding [[RemoveFile]] actions for the original file entries.
+ */
+ protected def writeFilesWithDVsOnDisk(
+ log: DeltaLog,
+ filesWithDVs: Seq[(AddFile, RoaringBitmapArray)]): Seq[Action] = {
+ val dvFileId = UUID.randomUUID()
+ withDVWriter(log, dvFileId) { writer =>
+ filesWithDVs.flatMap { case (currentFile, dv) =>
+ val range = writer.write(serializeRoaringBitmapArrayWithDefaultFormat(dv))
+ val dvData = DeletionVectorDescriptor.onDiskWithRelativePath(
+ id = dvFileId,
+ sizeInBytes = range.length,
+ cardinality = dv.cardinality,
+ offset = Some(range.offset))
+ val (add, remove) = currentFile.removeRows(
+ dvData,
+ updateStats = true
+ )
+ Seq(add, remove)
+ }
+ }
+ }
+
+ /**
+ * Removes the `numRowsToRemovePerFile` from each file via DV.
+ * Returns the total number of rows removed.
+ */
+ protected def removeRowsFromAllFilesInLog(
+ log: DeltaLog,
+ numRowsToRemovePerFile: Long): Long = {
+ var numFiles: Option[Int] = None
+ // This is needed to make the manual commit work correctly, since we are not actually
+ // running a command that produces metrics.
+ withSQLConf(DeltaSQLConf.DELTA_HISTORY_METRICS_ENABLED.key -> "false") {
+ val txn = log.startTransaction()
+ val allAddFiles = txn.snapshot.allFiles.collect()
+ numFiles = Some(allAddFiles.length)
+ val bitmap = RoaringBitmapArray(0L until numRowsToRemovePerFile: _*)
+ val actions = allAddFiles.flatMap { file =>
+ if (file.numPhysicalRecords.isDefined) {
+ // Only when stats are enabled. Can't check when stats are disabled
+ assert(file.numPhysicalRecords.get > numRowsToRemovePerFile)
+ }
+ writeFileWithDV(log, file, bitmap)
+ }
+ txn.commit(actions, DeltaOperations.Delete(predicate = Seq.empty))
+ }
+ numFiles.get * numRowsToRemovePerFile
+ }
+
+ def newDVStore(): DeletionVectorStore = {
+ // scalastyle:off deltahadoopconfiguration
+ DeletionVectorStore.createInstance(spark.sessionState.newHadoopConf())
+ // scalastyle:on deltahadoopconfiguration
+ }
+
+ /**
+ * Updates an [[AddFile]] with a [[DeletionVectorDescriptor]].
+ */
+ protected def updateFileDV(
+ addFile: AddFile,
+ dvDescriptor: DeletionVectorDescriptor): (AddFile, RemoveFile) = {
+ addFile.removeRows(
+ dvDescriptor,
+ updateStats = true
+ )
+ }
+
+ /** Delete the DV file in the given [[AddFile]]. Assumes the [[AddFile]] has a valid DV. */
+ protected def deleteDVFile(tablePath: String, addFile: AddFile): Unit = {
+ assert(addFile.deletionVector != null)
+ val dvPath = addFile.deletionVector.absolutePath(new Path(tablePath))
+ FileUtils.delete(new File(dvPath.toString))
+ }
+
+ /**
+ * Creates a [[DeletionVectorDescriptor]] from an [[RoaringBitmapArray]]
+ */
+ protected def writeDV(
+ log: DeltaLog,
+ bitmapArray: RoaringBitmapArray): DeletionVectorDescriptor = {
+ val dvFileId = UUID.randomUUID()
+ withDVWriter(log, dvFileId) { writer =>
+ val range = writer.write(serializeRoaringBitmapArrayWithDefaultFormat(bitmapArray))
+ DeletionVectorDescriptor.onDiskWithRelativePath(
+ id = dvFileId,
+ sizeInBytes = range.length,
+ cardinality = bitmapArray.cardinality,
+ offset = Some(range.offset))
+ }
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaColumnMappingTestUtils.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaColumnMappingTestUtils.scala
new file mode 100644
index 0000000000..68c47b42bb
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaColumnMappingTestUtils.scala
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{Column, Dataset}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.delta.actions.{Protocol, TableFeatureProtocolUtils}
+import org.apache.spark.sql.delta.schema.SchemaUtils
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaColumnMappingSelectedTestMixin
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{AtomicType, StructField, StructType}
+
+import org.apache.hadoop.fs.Path
+
+import java.io.File
+
+import scala.collection.mutable
+
+// spotless:off
+trait DeltaColumnMappingTestUtilsBase extends SharedSparkSession {
+
+ import testImplicits._
+
+ protected def columnMappingMode: String = NoMapping.name
+
+ private val PHYSICAL_NAME_REGEX =
+ "col-[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}".r
+
+ implicit class PhysicalNameString(s: String) {
+ def phy(deltaLog: DeltaLog): String = {
+ PHYSICAL_NAME_REGEX
+ .findFirstIn(s)
+ .getOrElse(getPhysicalName(s, deltaLog))
+ }
+ }
+
+ protected def columnMappingEnabled: Boolean = {
+ columnMappingModeString != "none"
+ }
+
+ protected def columnMappingModeString: String = {
+ spark.conf.getOption(DeltaConfigs.COLUMN_MAPPING_MODE.defaultTablePropertyKey)
+ .getOrElse("none")
+ }
+
+ /**
+ * Check if two schemas are equal ignoring column mapping metadata
+ * @param schema1 Schema
+ * @param schema2 Schema
+ */
+ protected def assertEqual(schema1: StructType, schema2: StructType): Unit = {
+ if (columnMappingEnabled) {
+ assert(
+ DeltaColumnMapping.dropColumnMappingMetadata(schema1) ==
+ DeltaColumnMapping.dropColumnMappingMetadata(schema2)
+ )
+ } else {
+ assert(schema1 == schema2)
+ }
+ }
+
+ /**
+ * Check if two table configurations are equal ignoring column mapping metadata
+ * @param config1 Table config
+ * @param config2 Table config
+ */
+ protected def assertEqual(
+ config1: Map[String, String],
+ config2: Map[String, String]): Unit = {
+ if (columnMappingEnabled) {
+ assert(dropColumnMappingConfigurations(config1) == dropColumnMappingConfigurations(config2))
+ } else {
+ assert(config1 == config2)
+ }
+ }
+
+ /**
+ * Check if a partition with specific values exists.
+ * Handles both column mapped and non-mapped cases
+ * @param partCol Partition column name
+ * @param partValue Partition value
+ * @param deltaLog DeltaLog
+ */
+ protected def assertPartitionWithValueExists(
+ partCol: String,
+ partValue: String,
+ deltaLog: DeltaLog): Unit = {
+ assert(getPartitionFilePathsWithValue(partCol, partValue, deltaLog).nonEmpty)
+ }
+
+ /**
+ * Assert partition exists in an array of set of partition names/paths
+ * @param partCol Partition column name
+ * @param deltaLog Delta log
+ * @param inputFiles Input files to scan for DF
+ */
+ protected def assertPartitionExists(
+ partCol: String,
+ deltaLog: DeltaLog,
+ inputFiles: Array[String]): Unit = {
+ val physicalName = partCol.phy(deltaLog)
+ val allFiles = deltaLog.update().allFiles.collect()
+ // NOTE: inputFiles are *not* URL-encoded.
+ val filesWithPartitions = inputFiles.map { f =>
+ allFiles.filter { af =>
+ f.contains(af.toPath.toString)
+ }.flatMap(_.partitionValues.keys).toSet
+ }
+ assert(filesWithPartitions.forall(p => p.count(_ == physicalName) > 0))
+ // for non-column mapped mode, we can check the file paths as well
+ if (!columnMappingEnabled) {
+ assert(inputFiles.forall(path => path.contains(s"$physicalName=")),
+ s"${inputFiles.toSeq.mkString("\n")}\ndidn't contain partition columns $physicalName")
+ }
+ }
+
+ /**
+ * Load Deltalog from path
+ * @param pathOrIdentifier Location
+ * @param isIdentifier Whether the previous argument is a metastore identifier
+ * @return
+ */
+ protected def loadDeltaLog(pathOrIdentifier: String, isIdentifier: Boolean = false): DeltaLog = {
+ if (isIdentifier) {
+ DeltaLog.forTable(spark, TableIdentifier(pathOrIdentifier))
+ } else {
+ DeltaLog.forTable(spark, pathOrIdentifier)
+ }
+ }
+
+ /**
+ * Convert a (nested) column string to sequence of name parts
+ * @param col Column string
+ * @return Sequence of parts
+ */
+ protected def columnNameToParts(col: String): Seq[String] = {
+ UnresolvedAttribute.parseAttributeName(col)
+ }
+
+ /**
+ * Get partition file paths for a specific partition value
+ * @param partCol Logical or physical partition name
+ * @param partValue Partition value
+ * @param deltaLog DeltaLog
+ * @return List of paths
+ */
+ protected def getPartitionFilePathsWithValue(
+ partCol: String,
+ partValue: String,
+ deltaLog: DeltaLog): Array[String] = {
+ getPartitionFilePaths(partCol, deltaLog).getOrElse(partValue, Array.empty)
+ }
+
+ /**
+ * Get the partition value for null
+ */
+ protected def nullPartitionValue: String = {
+ if (columnMappingEnabled) {
+ null
+ } else {
+ ExternalCatalogUtils.DEFAULT_PARTITION_NAME
+ }
+ }
+
+ /**
+ * Get partition file paths grouped by partition value
+ * @param partCol Logical or physical partition name
+ * @param deltaLog DeltaLog
+ * @return Partition value to paths
+ */
+ protected def getPartitionFilePaths(
+ partCol: String,
+ deltaLog: DeltaLog): Map[String, Array[String]] = {
+ if (columnMappingEnabled) {
+ val colName = partCol.phy(deltaLog)
+ deltaLog.update().allFiles.collect()
+ .groupBy(_.partitionValues(colName))
+ .mapValues(_.map(deltaLog.dataPath.toUri.getPath + "/" + _.path)).toMap
+ } else {
+ val partColEscaped = s"${ExternalCatalogUtils.escapePathName(partCol)}"
+ val dataPath = new File(deltaLog.dataPath.toUri.getPath)
+ dataPath.listFiles().filter(_.getName.startsWith(s"$partColEscaped="))
+ .groupBy(_.getName.split("=").last).mapValues(_.map(_.getPath)).toMap
+ }
+ }
+
+ /**
+ * Group a list of input file paths by partition key-value pair w.r.t. delta log
+ * @param inputFiles Input file paths
+ * @param deltaLog Delta log
+ * @return A mapped array each with the corresponding partition keys
+ */
+ protected def groupInputFilesByPartition(
+ inputFiles: Array[String],
+ deltaLog: DeltaLog): Map[(String, String), Array[String]] = {
+ if (columnMappingEnabled) {
+ val allFiles = deltaLog.update().allFiles.collect()
+ val grouped = inputFiles.flatMap { f =>
+ allFiles.find {
+ af => f.contains(af.toPath.toString)
+ }.head.partitionValues.map(entry => (f, entry))
+ }.groupBy(_._2)
+ grouped.mapValues(_.map(_._1)).toMap
+ } else {
+ inputFiles.groupBy(p => {
+ val nameParts = new Path(p).getParent.getName.split("=")
+ (nameParts(0), nameParts(1))
+ })
+ }
+ }
+
+ /**
+ * Drop column mapping configurations from Map
+ * @param configuration Table configuration
+ * @return Configuration
+ */
+ protected def dropColumnMappingConfigurations(
+ configuration: Map[String, String]): Map[String, String] = {
+ configuration - DeltaConfigs.COLUMN_MAPPING_MODE.key - DeltaConfigs.COLUMN_MAPPING_MAX_ID.key
+ }
+
+ /**
+ * Drop column mapping configurations from Dataset (e.g. sql("SHOW TBLPROPERTIES t1")
+ * @param configs Table configuration
+ * @return Configuration Dataset
+ */
+ protected def dropColumnMappingConfigurations(
+ configs: Dataset[(String, String)]): Dataset[(String, String)] = {
+ spark.createDataset(configs.collect().filter(p =>
+ !Seq(
+ DeltaConfigs.COLUMN_MAPPING_MAX_ID.key,
+ DeltaConfigs.COLUMN_MAPPING_MODE.key
+ ).contains(p._1)
+ ))
+ }
+
+ /** Return KV pairs of Protocol-related stuff for checking the result of DESCRIBE TABLE. */
+ protected def buildProtocolProps(snapshot: Snapshot): Seq[(String, String)] = {
+ val mergedConf =
+ DeltaConfigs.mergeGlobalConfigs(spark.sessionState.conf, snapshot.metadata.configuration)
+ val metadata = snapshot.metadata.copy(configuration = mergedConf)
+ var props = Seq(
+ (Protocol.MIN_READER_VERSION_PROP,
+ Protocol.forNewTable(spark, Some(metadata)).minReaderVersion.toString),
+ (Protocol.MIN_WRITER_VERSION_PROP,
+ Protocol.forNewTable(spark, Some(metadata)).minWriterVersion.toString))
+ if (snapshot.protocol.supportsReaderFeatures || snapshot.protocol.supportsWriterFeatures) {
+ props ++=
+ Protocol.minProtocolComponentsFromAutomaticallyEnabledFeatures(
+ spark, metadata, snapshot.protocol)
+ ._3
+ .map(f => (
+ s"${TableFeatureProtocolUtils.FEATURE_PROP_PREFIX}${f.name}",
+ TableFeatureProtocolUtils.FEATURE_PROP_SUPPORTED))
+ }
+ props
+ }
+
+ /**
+ * Convert (nested) column name string into physical name with reference from DeltaLog
+ * If target field does not have physical name, display name is returned
+ * @param col Logical column name
+ * @param deltaLog Reference DeltaLog
+ * @return Physical column name
+ */
+ protected def getPhysicalName(col: String, deltaLog: DeltaLog): String = {
+ val nameParts = UnresolvedAttribute.parseAttributeName(col)
+ val realSchema = deltaLog.update().schema
+ getPhysicalName(nameParts, realSchema)
+ }
+
+ protected def getPhysicalName(col: String, schema: StructType): String = {
+ val nameParts = UnresolvedAttribute.parseAttributeName(col)
+ getPhysicalName(nameParts, schema)
+ }
+
+ protected def getPhysicalName(nameParts: Seq[String], schema: StructType): String = {
+ SchemaUtils.findNestedFieldIgnoreCase(schema, nameParts, includeCollections = true)
+ .map(DeltaColumnMapping.getPhysicalName)
+ .get
+ }
+
+ protected def withColumnMappingConf(mode: String)(f: => Any): Any = {
+ withSQLConf(DeltaConfigs.COLUMN_MAPPING_MODE.defaultTablePropertyKey -> mode) {
+ f
+ }
+ }
+
+ protected def withMaxColumnIdConf(maxId: String)(f: => Any): Any = {
+ withSQLConf(DeltaConfigs.COLUMN_MAPPING_MAX_ID.defaultTablePropertyKey -> maxId) {
+ f
+ }
+ }
+
+ /**
+ * Gets the physical names of a path. This is used for converting column paths in stats schema,
+ * so it's ok to not support MapType and ArrayType.
+ */
+ def getPhysicalPathForStats(path: Seq[String], schema: StructType): Option[Seq[String]] = {
+ if (path.isEmpty) return Some(Seq.empty)
+ val field = schema.fields.find(_.name.equalsIgnoreCase(path.head))
+ field match {
+ case Some(f @ StructField(_, _: AtomicType, _, _ )) =>
+ if (path.size == 1) Some(Seq(DeltaColumnMapping.getPhysicalName(f))) else None
+ case Some(f @ StructField(_, st: StructType, _, _)) =>
+ val tail = getPhysicalPathForStats(path.tail, st)
+ tail.map(DeltaColumnMapping.getPhysicalName(f) +: _)
+ case _ =>
+ None
+ }
+ }
+
+ /**
+ * Convert (nested) column name string into physical name.
+ * Ignore parts of special paths starting with:
+ * 1. stats columns: minValues, maxValues, numRecords
+ * 2. stats df: stats_parsed
+ * 3. partition values: partitionValues_parsed, partitionValues
+ * @param col Logical column name (e.g. a.b.c)
+ * @param schema Reference schema with metadata
+ * @return Unresolved attribute with physical name paths
+ */
+ protected def convertColumnNameToAttributeWithPhysicalName(
+ col: String,
+ schema: StructType): UnresolvedAttribute = {
+ val parts = UnresolvedAttribute.parseAttributeName(col)
+ val shouldIgnoreFirstPart = Set(
+ "minValues",
+ "maxValues",
+ "numRecords",
+ Checkpoints.STRUCT_PARTITIONS_COL_NAME,
+ "partitionValues")
+ val shouldIgnoreSecondPart = Set(Checkpoints.STRUCT_STATS_COL_NAME, "stats")
+ val physical = if (shouldIgnoreFirstPart.contains(parts.head)) {
+ parts.head +: getPhysicalPathForStats(parts.tail, schema).getOrElse(parts.tail)
+ } else if (shouldIgnoreSecondPart.contains(parts.head)) {
+ parts.take(2) ++ getPhysicalPathForStats(parts.slice(2, parts.length), schema)
+ .getOrElse(parts.slice(2, parts.length))
+ } else {
+ getPhysicalPathForStats(parts, schema).getOrElse(parts)
+ }
+ UnresolvedAttribute(physical)
+ }
+
+ /**
+ * Convert a list of (nested) stats columns into physical name with reference from DeltaLog
+ * @param columns Logical columns
+ * @param deltaLog Reference DeltaLog
+ * @return Physical columns
+ */
+ protected def convertToPhysicalColumns(
+ columns: Seq[Column],
+ deltaLog: DeltaLog): Seq[Column] = {
+ val schema = deltaLog.update().schema
+ columns.map { col =>
+ val newExpr = col.expr.transform {
+ case a: Attribute =>
+ convertColumnNameToAttributeWithPhysicalName(a.name, schema)
+ }
+ Column(newExpr)
+ }
+ }
+
+ /**
+ * Standard CONVERT TO DELTA
+ * @param tableOrPath String
+ */
+ protected def convertToDelta(tableOrPath: String): Unit = {
+ sql(s"CONVERT TO DELTA $tableOrPath")
+ }
+
+ /**
+ * Force enable streaming read (with possible data loss) on column mapping enabled table with
+ * drop / rename schema changes.
+ */
+ protected def withStreamingReadOnColumnMappingTableEnabled(f: => Unit): Unit = {
+ if (columnMappingEnabled) {
+ withSQLConf(DeltaSQLConf
+ .DELTA_STREAMING_UNSAFE_READ_ON_INCOMPATIBLE_COLUMN_MAPPING_SCHEMA_CHANGES.key -> "true") {
+ f
+ }
+ } else {
+ f
+ }
+ }
+
+}
+
+trait DeltaColumnMappingTestUtils extends DeltaColumnMappingTestUtilsBase
+
+/**
+ * Include this trait to enable Id column mapping mode for a suite
+ */
+trait DeltaColumnMappingEnableIdMode extends SharedSparkSession
+ with DeltaColumnMappingTestUtils
+ with DeltaColumnMappingSelectedTestMixin {
+
+ protected override def columnMappingMode: String = IdMapping.name
+
+ protected override def sparkConf: SparkConf =
+ super.sparkConf.set(DeltaConfigs.COLUMN_MAPPING_MODE.defaultTablePropertyKey, "id")
+
+ /**
+ * CONVERT TO DELTA blocked in id mode
+ */
+ protected override def convertToDelta(tableOrPath: String): Unit =
+ throw DeltaErrors.convertToDeltaWithColumnMappingNotSupported(
+ DeltaColumnMappingMode(columnMappingModeString)
+ )
+}
+
+/**
+ * Include this trait to enable Name column mapping mode for a suite
+ */
+trait DeltaColumnMappingEnableNameMode extends SharedSparkSession
+ with DeltaColumnMappingTestUtils
+ with DeltaColumnMappingSelectedTestMixin {
+
+ protected override def columnMappingMode: String = NameMapping.name
+
+ protected override def sparkConf: SparkConf =
+ super.sparkConf.set(DeltaConfigs.COLUMN_MAPPING_MODE.defaultTablePropertyKey, columnMappingMode)
+
+ /**
+ * CONVERT TO DELTA can be possible under name mode in tests
+ */
+ protected override def convertToDelta(tableOrPath: String): Unit = {
+ withColumnMappingConf("none") {
+ super.convertToDelta(tableOrPath)
+ }
+
+ val (deltaPath, deltaLog) =
+ if (tableOrPath.contains("parquet") && tableOrPath.contains("`")) {
+ // parquet.`PATH`
+ val plainPath = tableOrPath.split('.').last.drop(1).dropRight(1)
+ (s"delta.`$plainPath`", DeltaLog.forTable(spark, plainPath))
+ } else {
+ (tableOrPath, DeltaLog.forTable(spark, TableIdentifier(tableOrPath)))
+ }
+
+ val tableReaderVersion = deltaLog.unsafeVolatileSnapshot.protocol.minReaderVersion
+ val tableWriterVersion = deltaLog.unsafeVolatileSnapshot.protocol.minWriterVersion
+ val requiredReaderVersion = if (tableWriterVersion >=
+ TableFeatureProtocolUtils.TABLE_FEATURES_MIN_WRITER_VERSION) {
+ // If the writer version of the table supports table features, we need to
+ // bump the reader version to table features to enable column mapping.
+ TableFeatureProtocolUtils.TABLE_FEATURES_MIN_READER_VERSION
+ } else {
+ ColumnMappingTableFeature.minReaderVersion
+ }
+ val readerVersion = spark.conf.get(DeltaSQLConf.DELTA_PROTOCOL_DEFAULT_READER_VERSION).max(
+ requiredReaderVersion)
+ val writerVersion = spark.conf.get(DeltaSQLConf.DELTA_PROTOCOL_DEFAULT_WRITER_VERSION).max(
+ ColumnMappingTableFeature.minWriterVersion)
+
+ val properties = mutable.ListBuffer(DeltaConfigs.COLUMN_MAPPING_MODE.key -> "name")
+ if (tableReaderVersion < readerVersion) {
+ properties += DeltaConfigs.MIN_READER_VERSION.key -> readerVersion.toString
+ }
+ if (tableWriterVersion < writerVersion) {
+ properties += DeltaConfigs.MIN_WRITER_VERSION.key -> writerVersion.toString
+ }
+ val propertiesStr = properties.map(kv => s"'${kv._1}' = '${kv._2}'").mkString(", ")
+ sql(s"ALTER TABLE $deltaPath SET TBLPROPERTIES ($propertiesStr)")
+ }
+
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaTestUtils.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaTestUtils.scala
new file mode 100644
index 0000000000..4e7326c8c1
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/DeltaTestUtils.scala
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta
+
+import org.apache.spark.{SparkContext, SparkFunSuite, SparkThrowable}
+import org.apache.spark.scheduler.{JobFailed, SparkListener, SparkListenerJobEnd, SparkListenerJobStart}
+import org.apache.spark.sql.{AnalysisException, DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.util.{quietly, FailFastMode}
+import org.apache.spark.sql.delta.DeltaTestUtils.Plans
+import org.apache.spark.sql.delta.actions._
+import org.apache.spark.sql.delta.commands.cdc.CDCReader
+import org.apache.spark.sql.delta.sources.DeltaSQLConf
+import org.apache.spark.sql.delta.test.DeltaSQLTestUtils
+import org.apache.spark.sql.delta.util.FileNames
+import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.aggregate.HashAggregateExec
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.QueryExecutionListener
+import org.apache.spark.util.Utils
+
+import com.databricks.spark.util.{Log4jUsageLogger, UsageRecord}
+import com.fasterxml.jackson.databind.ObjectMapper
+import com.fasterxml.jackson.module.scala.DefaultScalaModule
+import io.delta.tables.{DeltaTable => IODeltaTable}
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.scalatest.BeforeAndAfterEach
+
+import java.io.{BufferedReader, File, InputStreamReader}
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Locale
+import java.util.concurrent.ConcurrentHashMap
+
+import scala.collection.JavaConverters._
+import scala.collection.concurrent
+import scala.reflect.ClassTag
+import scala.util.matching.Regex
+
+// spotless:off
+trait DeltaTestUtilsBase {
+ import DeltaTestUtils.TableIdentifierOrPath
+
+ final val BOOLEAN_DOMAIN: Seq[Boolean] = Seq(true, false)
+
+ class PlanCapturingListener() extends QueryExecutionListener {
+
+ private[this] var capturedPlans = List.empty[Plans]
+
+ def plans: Seq[Plans] = capturedPlans.reverse
+
+ override def onSuccess(funcName: String, qe: QueryExecution, durationNs: Long): Unit = {
+ capturedPlans ::= Plans(
+ qe.analyzed,
+ qe.optimizedPlan,
+ qe.sparkPlan,
+ qe.executedPlan)
+ }
+
+ override def onFailure(
+ funcName: String, qe: QueryExecution, error: Exception): Unit = {}
+ }
+
+ /**
+ * Run a thunk with physical plans for all queries captured and passed into a provided buffer.
+ */
+ def withLogicalPlansCaptured[T](
+ spark: SparkSession,
+ optimizedPlan: Boolean)(
+ thunk: => Unit): Seq[LogicalPlan] = {
+ val planCapturingListener = new PlanCapturingListener
+
+ spark.sparkContext.listenerBus.waitUntilEmpty(15000)
+ spark.listenerManager.register(planCapturingListener)
+ try {
+ thunk
+ spark.sparkContext.listenerBus.waitUntilEmpty(15000)
+ planCapturingListener.plans.map { plans =>
+ if (optimizedPlan) plans.optimized else plans.analyzed
+ }
+ } finally {
+ spark.listenerManager.unregister(planCapturingListener)
+ }
+ }
+
+ /**
+ * Run a thunk with physical plans for all queries captured and passed into a provided buffer.
+ */
+ def withPhysicalPlansCaptured[T](
+ spark: SparkSession)(
+ thunk: => Unit): Seq[SparkPlan] = {
+ val planCapturingListener = new PlanCapturingListener
+
+ spark.sparkContext.listenerBus.waitUntilEmpty(15000)
+ spark.listenerManager.register(planCapturingListener)
+ try {
+ thunk
+ spark.sparkContext.listenerBus.waitUntilEmpty(15000)
+ planCapturingListener.plans.map(_.sparkPlan)
+ } finally {
+ spark.listenerManager.unregister(planCapturingListener)
+ }
+ }
+
+ /**
+ * Run a thunk with logical and physical plans for all queries captured and passed
+ * into a provided buffer.
+ */
+ def withAllPlansCaptured[T](
+ spark: SparkSession)(
+ thunk: => Unit): Seq[Plans] = {
+ val planCapturingListener = new PlanCapturingListener
+
+ spark.sparkContext.listenerBus.waitUntilEmpty(15000)
+ spark.listenerManager.register(planCapturingListener)
+ try {
+ thunk
+ spark.sparkContext.listenerBus.waitUntilEmpty(15000)
+ planCapturingListener.plans
+ } finally {
+ spark.listenerManager.unregister(planCapturingListener)
+ }
+ }
+
+ def countSparkJobs(sc: SparkContext, f: => Unit): Int = {
+ val jobs: concurrent.Map[Int, Long] = new ConcurrentHashMap[Int, Long]().asScala
+ val listener = new SparkListener {
+ override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
+ jobs.put(jobStart.jobId, jobStart.stageInfos.map(_.numTasks).sum)
+ }
+ override def onJobEnd(jobEnd: SparkListenerJobEnd): Unit = jobEnd.jobResult match {
+ case JobFailed(_) => jobs.remove(jobEnd.jobId)
+ case _ => // On success, do nothing.
+ }
+ }
+ sc.addSparkListener(listener)
+ try {
+ sc.listenerBus.waitUntilEmpty(15000)
+ f
+ sc.listenerBus.waitUntilEmpty(15000)
+ } finally {
+ sc.removeSparkListener(listener)
+ }
+ // Spark will always log a job start/end event even when the job does not launch any task.
+ jobs.values.count(_ > 0)
+ }
+
+ /** Filter `usageRecords` by the `opType` tag or field. */
+ def filterUsageRecords(usageRecords: Seq[UsageRecord], opType: String): Seq[UsageRecord] = {
+ usageRecords.filter { r =>
+ r.tags.get("opType").contains(opType) || r.opType.map(_.typeName).contains(opType)
+ }
+ }
+
+ def collectUsageLogs(opType: String)(f: => Unit): collection.Seq[UsageRecord] = {
+ Log4jUsageLogger.track(f).filter { r =>
+ r.metric == "tahoeEvent" &&
+ r.tags.get("opType").contains(opType)
+ }
+ }
+
+ /**
+ * Remove protocol and metadata fields from checksum file of json format
+ */
+ def removeProtocolAndMetadataFromChecksumFile(checksumFilePath : Path): Unit = {
+ // scalastyle:off deltahadoopconfiguration
+ val fs = checksumFilePath.getFileSystem(
+ SparkSession.getActiveSession.map(_.sessionState.newHadoopConf()).get
+ )
+ // scalastyle:on deltahadoopconfiguration
+ if (!fs.exists(checksumFilePath)) return
+ val stream = fs.open(checksumFilePath)
+ val reader = new BufferedReader(new InputStreamReader(stream, UTF_8))
+ val content = reader.readLine()
+ stream.close()
+ val mapper = new ObjectMapper()
+ mapper.registerModule(DefaultScalaModule)
+ val map = mapper.readValue(content, classOf[Map[String, String]])
+ val partialContent = mapper.writeValueAsString(map.-("protocol").-("metadata")) + "\n"
+ val output = fs.create(checksumFilePath, true)
+ output.write(partialContent.getBytes(UTF_8))
+ output.close()
+ }
+
+ protected def getfindTouchedFilesJobPlans(plans: Seq[Plans]): SparkPlan = {
+ // The expected plan for touched file computation is of the format below.
+ // The data column should be pruned from both leaves.
+ // HashAggregate(output=[count#3463L])
+ // +- HashAggregate(output=[count#3466L])
+ // +- Project
+ // +- Filter (isnotnull(count#3454L) AND (count#3454L > 1))
+ // +- HashAggregate(output=[count#3454L])
+ // +- HashAggregate(output=[_row_id_#3418L, sum#3468L])
+ // +- Project [_row_id_#3418L, UDF(_file_name_#3422) AS one#3448]
+ // +- BroadcastHashJoin [id#3342L], [id#3412L], Inner, BuildLeft
+ // :- Project [id#3342L]
+ // : +- Filter isnotnull(id#3342L)
+ // : +- FileScan parquet [id#3342L,part#3343L]
+ // +- Filter isnotnull(id#3412L)
+ // +- Project [...]
+ // +- Project [...]
+ // +- FileScan parquet [id#3412L,part#3413L]
+ // Note: It can be RDDScanExec instead of FileScan if the source was materialized.
+ // We pick the first plan starting from FileScan and ending in HashAggregate as a
+ // stable heuristic for the one we want.
+ plans.map(_.executedPlan)
+ .filter {
+ case WholeStageCodegenExec(hash: HashAggregateExec) =>
+ hash.collectLeaves().size == 2 &&
+ hash.collectLeaves()
+ .forall { s =>
+ s.isInstanceOf[FileSourceScanExec] ||
+ s.isInstanceOf[RDDScanExec]
+ }
+ case _ => false
+ }.head
+ }
+
+ /**
+ * Separate name- from path-based SQL table identifiers.
+ */
+ def getTableIdentifierOrPath(sqlIdentifier: String): TableIdentifierOrPath = {
+ // Match: delta.`path`[[ as] alias] or tahoe.`path`[[ as] alias]
+ val pathMatcher: Regex = raw"(?:delta|tahoe)\.`([^`]+)`(?:(?: as)? (.+))?".r
+ // Match: db.table[[ as] alias]
+ val qualifiedDbMatcher: Regex = raw"`?([^\.` ]+)`?\.`?([^\.` ]+)`?(?:(?: as)? (.+))?".r
+ // Match: table[[ as] alias]
+ val unqualifiedNameMatcher: Regex = raw"([^ ]+)(?:(?: as)? (.+))?".r
+ sqlIdentifier match {
+ case pathMatcher(path, alias) =>
+ TableIdentifierOrPath.Path(path, Option(alias))
+ case qualifiedDbMatcher(dbName, tableName, alias) =>
+ TableIdentifierOrPath.Identifier(TableIdentifier(tableName, Some(dbName)), Option(alias))
+ case unqualifiedNameMatcher(tableName, alias) =>
+ TableIdentifierOrPath.Identifier(TableIdentifier(tableName), Option(alias))
+ }
+ }
+
+ /**
+ * Produce a DeltaTable instance given a `TableIdentifierOrPath` instance.
+ */
+ def getDeltaTableForIdentifierOrPath(
+ spark: SparkSession,
+ identifierOrPath: TableIdentifierOrPath): IODeltaTable = {
+ identifierOrPath match {
+ case TableIdentifierOrPath.Identifier(id, optionalAlias) =>
+ val table = IODeltaTable.forName(spark, id.unquotedString)
+ optionalAlias.map(table.as(_)).getOrElse(table)
+ case TableIdentifierOrPath.Path(path, optionalAlias) =>
+ val table = IODeltaTable.forPath(spark, path)
+ optionalAlias.map(table.as(_)).getOrElse(table)
+ }
+ }
+
+ @deprecated("Use checkError() instead")
+ protected def errorContains(errMsg: String, str: String): Unit = {
+ assert(errMsg.toLowerCase(Locale.ROOT).contains(str.toLowerCase(Locale.ROOT)))
+ }
+
+ /**
+ * Helper types to define the expected result of a test case.
+ * Either:
+ * - Success: include an expected value to check, e.g. expected schema or result as a DF or rows.
+ * - Failure: an exception is thrown and the caller passes a function to check that it matches an
+ * expected error, typ. `checkError()` or `checkErrorMatchPVals()`.
+ */
+ sealed trait ExpectedResult[-T]
+ object ExpectedResult {
+ case class Success[T](expected: T) extends ExpectedResult[T]
+ case class Failure[T](checkError: SparkThrowable => Unit) extends ExpectedResult[T]
+ }
+
+ /** Utility method to check exception `e` is of type `E` or a cause of it is of type `E` */
+ def findIfResponsible[E <: Throwable: ClassTag](e: Throwable): Option[E] = e match {
+ case culprit: E => Some(culprit)
+ case _ =>
+ val children = Option(e.getCause).iterator ++ e.getSuppressed.iterator
+ children
+ .map(findIfResponsible[E](_))
+ .collectFirst { case Some(culprit) => culprit }
+ }
+
+ def verifyBackfilled(file: FileStatus): Unit = {
+ val unbackfilled = file.getPath.getName.matches(FileNames.uuidDeltaFileRegex.toString)
+ assert(!unbackfilled, s"File $file was not backfilled")
+ }
+
+ def verifyUnbackfilled(file: FileStatus): Unit = {
+ val unbackfilled = file.getPath.getName.matches(FileNames.uuidDeltaFileRegex.toString)
+ assert(unbackfilled, s"File $file was backfilled")
+ }
+}
+
+trait DeltaCheckpointTestUtils
+ extends DeltaTestUtilsBase { self: SparkFunSuite with SharedSparkSession =>
+
+ def testDifferentCheckpoints(testName: String, quiet: Boolean = false)
+ (f: (CheckpointPolicy.Policy, Option[V2Checkpoint.Format]) => Unit): Unit = {
+ test(s"$testName [Checkpoint V1]") {
+ def testFunc(): Unit = {
+ withSQLConf(DeltaConfigs.CHECKPOINT_POLICY.defaultTablePropertyKey ->
+ CheckpointPolicy.Classic.name) {
+ f(CheckpointPolicy.Classic, None)
+ }
+ }
+ if (quiet) quietly { testFunc() } else testFunc()
+ }
+ for (checkpointFormat <- V2Checkpoint.Format.ALL)
+ test(s"$testName [Checkpoint V2, format: ${checkpointFormat.name}]") {
+ def testFunc(): Unit = {
+ withSQLConf(
+ DeltaConfigs.CHECKPOINT_POLICY.defaultTablePropertyKey -> CheckpointPolicy.V2.name,
+ DeltaSQLConf.CHECKPOINT_V2_TOP_LEVEL_FILE_FORMAT.key -> checkpointFormat.name
+ ) {
+ f(CheckpointPolicy.V2, Some(checkpointFormat))
+ }
+ }
+ if (quiet) quietly { testFunc() } else testFunc()
+ }
+ }
+
+ /**
+ * Helper method to get the dataframe corresponding to the files which has the file actions for a
+ * given checkpoint.
+ */
+ def getCheckpointDfForFilesContainingFileActions(
+ log: DeltaLog,
+ checkpointFile: Path): DataFrame = {
+ val ci = CheckpointInstance.apply(checkpointFile)
+ val allCheckpointFiles = log
+ .listFrom(ci.version)
+ .filter(FileNames.isCheckpointFile)
+ .filter(f => CheckpointInstance(f.getPath) == ci)
+ .toSeq
+ val fileActionsFileIndex = ci.format match {
+ case CheckpointInstance.Format.V2 =>
+ val incompleteCheckpointProvider = ci.getCheckpointProvider(log, allCheckpointFiles)
+ val df = log.loadIndex(incompleteCheckpointProvider.topLevelFileIndex.get, Action.logSchema)
+ val sidecarFileStatuses = df.as[SingleAction].collect().map(_.unwrap).collect {
+ case sf: SidecarFile => sf
+ }.map(sf => sf.toFileStatus(log.logPath))
+ DeltaLogFileIndex(DeltaLogFileIndex.CHECKPOINT_FILE_FORMAT_PARQUET, sidecarFileStatuses)
+ case CheckpointInstance.Format.SINGLE | CheckpointInstance.Format.WITH_PARTS =>
+ DeltaLogFileIndex(DeltaLogFileIndex.CHECKPOINT_FILE_FORMAT_PARQUET,
+ allCheckpointFiles.toArray)
+ case _ =>
+ throw new Exception(s"Unexpected checkpoint format for file $checkpointFile")
+ }
+ fileActionsFileIndex.files
+ .map(fileStatus => spark.read.parquet(fileStatus.getPath.toString))
+ .reduce(_.union(_))
+ }
+}
+
+object DeltaTestUtils extends DeltaTestUtilsBase {
+
+ sealed trait TableIdentifierOrPath
+ object TableIdentifierOrPath {
+ case class Identifier(id: TableIdentifier, alias: Option[String])
+ extends TableIdentifierOrPath
+ case class Path(path: String, alias: Option[String]) extends TableIdentifierOrPath
+ }
+
+ case class Plans(
+ analyzed: LogicalPlan,
+ optimized: LogicalPlan,
+ sparkPlan: SparkPlan,
+ executedPlan: SparkPlan)
+
+ /**
+ * Creates an AddFile that can be used for tests where the exact parameters do not matter.
+ */
+ def createTestAddFile(
+ encodedPath: String = "foo",
+ partitionValues: Map[String, String] = Map.empty,
+ size: Long = 1L,
+ modificationTime: Long = 1L,
+ dataChange: Boolean = true,
+ stats: String = "{\"numRecords\": 1}"): AddFile = {
+ AddFile(encodedPath, partitionValues, size, modificationTime, dataChange, stats)
+ }
+
+ /**
+ * Extracts the table name and alias (if any) from the given string. Correctly handles whitespaces
+ * in table name but doesn't support whitespaces in alias.
+ */
+ def parseTableAndAlias(table: String): (String, Option[String]) = {
+ // Matches 'delta.`path` AS alias' (case insensitive).
+ val deltaPathWithAsAlias = raw"(?i)(delta\.`.+`)(?: AS) (\S+)".r
+ // Matches 'delta.`path` alias'.
+ val deltaPathWithAlias = raw"(delta\.`.+`) (\S+)".r
+ // Matches 'delta.`path`'.
+ val deltaPath = raw"(delta\.`.+`)".r
+ // Matches 'tableName AS alias' (case insensitive).
+ val tableNameWithAsAlias = raw"(?i)(.+)(?: AS) (\S+)".r
+ // Matches 'tableName alias'.
+ val tableNameWithAlias = raw"(.+) (.+)".r
+
+ table match {
+ case deltaPathWithAsAlias(tableName, alias) => tableName -> Some(alias)
+ case deltaPathWithAlias(tableName, alias) => tableName -> Some(alias)
+ case deltaPath(tableName) => tableName -> None
+ case tableNameWithAsAlias(tableName, alias) => tableName -> Some(alias)
+ case tableNameWithAlias(tableName, alias) => tableName -> Some(alias)
+ case tableName => tableName -> None
+ }
+ }
+
+ /**
+ * Implements an ordering where `x < y` iff both reader and writer versions of
+ * `x` are strictly less than those of `y`.
+ *
+ * Can be used to conveniently check that this relationship holds in tests/assertions
+ * without having to write out the conjunction of the two subconditions every time.
+ */
+ case object StrictProtocolOrdering extends PartialOrdering[Protocol] {
+ override def tryCompare(x: Protocol, y: Protocol): Option[Int] = {
+ if (x.minReaderVersion == y.minReaderVersion &&
+ x.minWriterVersion == y.minWriterVersion) {
+ Some(0)
+ } else if (x.minReaderVersion < y.minReaderVersion &&
+ x.minWriterVersion < y.minWriterVersion) {
+ Some(-1)
+ } else if (x.minReaderVersion > y.minReaderVersion &&
+ x.minWriterVersion > y.minWriterVersion) {
+ Some(1)
+ } else {
+ None
+ }
+ }
+
+ override def lteq(x: Protocol, y: Protocol): Boolean =
+ x.minReaderVersion <= y.minReaderVersion && x.minWriterVersion <= y.minWriterVersion
+
+ // Just a more readable version of `lteq`.
+ def fulfillsVersionRequirements(actual: Protocol, requirement: Protocol): Boolean =
+ lteq(requirement, actual)
+ }
+}
+
+trait DeltaTestUtilsForTempViews
+ extends SharedSparkSession
+ with DeltaTestUtilsBase {
+
+ def testWithTempView(testName: String)(testFun: Boolean => Any): Unit = {
+ Seq(true, false).foreach { isSQLTempView =>
+ val tempViewUsed = if (isSQLTempView) "SQL TempView" else "Dataset TempView"
+ test(s"$testName - $tempViewUsed") {
+ withTempView("v") {
+ testFun(isSQLTempView)
+ }
+ }
+ }
+ }
+
+ def testQuietlyWithTempView(testName: String)(testFun: Boolean => Any): Unit = {
+ Seq(true, false).foreach { isSQLTempView =>
+ val tempViewUsed = if (isSQLTempView) "SQL TempView" else "Dataset TempView"
+ testQuietly(s"$testName - $tempViewUsed") {
+ withTempView("v") {
+ testFun(isSQLTempView)
+ }
+ }
+ }
+ }
+
+ def createTempViewFromTable(
+ tableName: String,
+ isSQLTempView: Boolean,
+ format: Option[String] = None): Unit = {
+ if (isSQLTempView) {
+ sql(s"CREATE OR REPLACE TEMP VIEW v AS SELECT * from $tableName")
+ } else {
+ spark.read.format(format.getOrElse("delta")).table(tableName).createOrReplaceTempView("v")
+ }
+ }
+
+ def createTempViewFromSelect(text: String, isSQLTempView: Boolean): Unit = {
+ if (isSQLTempView) {
+ sql(s"CREATE OR REPLACE TEMP VIEW v AS $text")
+ } else {
+ sql(text).createOrReplaceTempView("v")
+ }
+ }
+
+ def testErrorMessageAndClass(
+ isSQLTempView: Boolean,
+ ex: AnalysisException,
+ expectedErrorMsgForSQLTempView: String = null,
+ expectedErrorMsgForDataSetTempView: String = null,
+ expectedErrorClassForSQLTempView: String = null,
+ expectedErrorClassForDataSetTempView: String = null): Unit = {
+ if (isSQLTempView) {
+ if (expectedErrorMsgForSQLTempView != null) {
+ errorContains(ex.getMessage, expectedErrorMsgForSQLTempView)
+ }
+ if (expectedErrorClassForSQLTempView != null) {
+ assert(ex.getErrorClass == expectedErrorClassForSQLTempView)
+ }
+ } else {
+ if (expectedErrorMsgForDataSetTempView != null) {
+ errorContains(ex.getMessage, expectedErrorMsgForDataSetTempView)
+ }
+ if (expectedErrorClassForDataSetTempView != null) {
+ assert(ex.getErrorClass == expectedErrorClassForDataSetTempView, ex.getMessage)
+ }
+ }
+ }
+}
+
+/**
+ * Trait collecting helper methods for DML tests e.p. creating a test table for each test and
+ * cleaning it up after each test.
+ */
+trait DeltaDMLTestUtils
+ extends DeltaSQLTestUtils
+ with DeltaTestUtilsBase
+ with BeforeAndAfterEach {
+ self: SharedSparkSession =>
+
+ import testImplicits._
+
+ protected var tempDir: File = _
+
+ protected var deltaLog: DeltaLog = _
+
+ protected def tempPath: String = tempDir.getCanonicalPath
+
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ // Using a space in path to provide coverage for special characters.
+ tempDir = Utils.createTempDir(namePrefix = "spark test")
+ deltaLog = DeltaLog.forTable(spark, new Path(tempPath))
+ }
+
+ override protected def afterEach(): Unit = {
+ try {
+ Utils.deleteRecursively(tempDir)
+ DeltaLog.clearCache()
+ } finally {
+ super.afterEach()
+ }
+ }
+
+ protected def append(df: DataFrame, partitionBy: Seq[String] = Nil): Unit = {
+ val dfw = df.write.format("delta").mode("append")
+ if (partitionBy.nonEmpty) {
+ dfw.partitionBy(partitionBy: _*)
+ }
+ dfw.save(tempPath)
+ }
+
+ protected def withKeyValueData(
+ source: Seq[(Int, Int)],
+ target: Seq[(Int, Int)],
+ isKeyPartitioned: Boolean = false,
+ sourceKeyValueNames: (String, String) = ("key", "value"),
+ targetKeyValueNames: (String, String) = ("key", "value"))(
+ thunk: (String, String) => Unit = null): Unit = {
+
+ import testImplicits._
+
+ append(target.toDF(targetKeyValueNames._1, targetKeyValueNames._2).coalesce(2),
+ if (isKeyPartitioned) Seq(targetKeyValueNames._1) else Nil)
+ withTempView("source") {
+ source.toDF(sourceKeyValueNames._1, sourceKeyValueNames._2).createOrReplaceTempView("source")
+ thunk("source", s"delta.`$tempPath`")
+ }
+ }
+
+ /**
+ * Parse the input JSON data into a dataframe, one row per input element.
+ * Throws an exception on malformed inputs or records that don't comply with the provided schema.
+ */
+ protected def readFromJSON(data: Seq[String], schema: StructType = null): DataFrame = {
+ if (schema != null) {
+ spark.read
+ .schema(schema)
+ .option("mode", FailFastMode.name)
+ .json(data.toDS)
+ } else {
+ spark.read
+ .option("mode", FailFastMode.name)
+ .json(data.toDS)
+ }
+ }
+
+ protected def readDeltaTable(path: String): DataFrame = {
+ spark.read.format("delta").load(path)
+ }
+
+ protected def getDeltaFileStmt(path: String): String = s"SELECT * FROM delta.`$path`"
+
+ /**
+ * Finds the latest operation of the given type that ran on the test table and returns the
+ * dataframe with the changes of the corresponding table version.
+ *
+ * @param operation Delta operation name, see [[DeltaOperations]].
+ */
+ protected def getCDCForLatestOperation(deltaLog: DeltaLog, operation: String): DataFrame = {
+ val latestOperation = deltaLog.history
+ .getHistory(None)
+ .find(_.operation == operation)
+ assert(latestOperation.nonEmpty, s"Couldn't find a ${operation} operation to check CDF")
+
+ val latestOperationVersion = latestOperation.get.version
+ assert(latestOperationVersion.nonEmpty,
+ s"Latest ${operation} operation doesn't have a version associated with it")
+
+ CDCReader
+ .changesToBatchDF(
+ deltaLog,
+ latestOperationVersion.get,
+ latestOperationVersion.get,
+ spark)
+ .drop(CDCReader.CDC_COMMIT_TIMESTAMP)
+ .drop(CDCReader.CDC_COMMIT_VERSION)
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaColumnMappingSelectedTestMixin.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaColumnMappingSelectedTestMixin.scala
new file mode 100644
index 0000000000..135dd97bfa
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaColumnMappingSelectedTestMixin.scala
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta.test
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.delta.DeltaColumnMappingTestUtils
+import org.apache.spark.sql.delta.DeltaConfigs
+
+import org.scalactic.source.Position
+import org.scalatest.Tag
+import org.scalatest.exceptions.TestFailedException
+
+import scala.collection.mutable
+
+// spotless:off
+/**
+ * A trait for selective enabling certain tests to run for column mapping modes
+ */
+trait DeltaColumnMappingSelectedTestMixin extends SparkFunSuite
+ with DeltaSQLTestUtils with DeltaColumnMappingTestUtils {
+
+ protected def runOnlyTests: Seq[String] = Seq()
+
+ /**
+ * If true, will run all tests.
+ * Requires that `runOnlyTests` is empty.
+ */
+ protected def runAllTests: Boolean = false
+
+ private val testsRun: mutable.Set[String] = mutable.Set.empty
+
+ override protected def test(
+ testName: String,
+ testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit = {
+ require(!runAllTests || runOnlyTests.isEmpty,
+ "If `runAllTests` is true then `runOnlyTests` must be empty")
+
+ if (runAllTests || runOnlyTests.contains(testName)) {
+ super.test(s"$testName - column mapping $columnMappingMode mode", testTags: _*) {
+ testsRun.add(testName)
+ withSQLConf(
+ DeltaConfigs.COLUMN_MAPPING_MODE.defaultTablePropertyKey -> columnMappingMode) {
+ testFun
+ }
+ }
+ } else {
+ super.ignore(s"$testName - ignored by DeltaColumnMappingSelectedTestMixin")(testFun)
+ }
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ val missingTests = runOnlyTests.toSet diff testsRun
+ if (missingTests.nonEmpty) {
+ throw new TestFailedException(
+ Some("Not all selected column mapping tests were run. Missing: " +
+ missingTests.mkString(", ")), None, 0)
+ }
+ }
+
+}
+// spotless:on
diff --git a/gluten-ut/common/src/test/scala/org/apache/gluten/utils/BackendTestUtils.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaExcludedTestMixin.scala
similarity index 57%
copy from gluten-ut/common/src/test/scala/org/apache/gluten/utils/BackendTestUtils.scala
copy to backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaExcludedTestMixin.scala
index 7635316610..b166697284 100644
--- a/gluten-ut/common/src/test/scala/org/apache/gluten/utils/BackendTestUtils.scala
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaExcludedTestMixin.scala
@@ -14,20 +14,27 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.gluten.utils
+package org.apache.spark.sql.delta.test
-import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.spark.sql.QueryTest
-// FIXME: We mixed the use of "Utils" and "Util" in utility class naming
-// FIXME: The utility methods "is...BackendLoaded" stands because we rely
-// on legacy backend-specific code in UTs. The new usages of the methods should
-// be better avoided. We'd remove them until the time is ripe.
-object BackendTestUtils {
- def isVeloxBackendLoaded(): Boolean = {
- BackendsApiManager.getBackendName == "velox" // the same with VeloxBackend.BACKEND_NAME
- }
+import org.scalactic.source.Position
+import org.scalatest.Tag
+
+// spotless:off
+trait DeltaExcludedTestMixin extends QueryTest {
+
+ /** Tests to be ignored by the runner. */
+ override def excluded: Seq[String] = Seq.empty
- def isCHBackendLoaded(): Boolean = {
- BackendsApiManager.getBackendName == "ch" // the same with CHBackend.BACKEND_NAME
+ protected override def test(testName: String, testTags: Tag*)
+ (testFun: => Any)
+ (implicit pos: Position): Unit = {
+ if (excluded.contains(testName)) {
+ super.ignore(testName, testTags: _*)(testFun)
+ } else {
+ super.test(testName, testTags: _*)(testFun)
+ }
}
}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala
new file mode 100644
index 0000000000..3d94d2bde3
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta.test
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.delta.catalog.DeltaCatalog
+import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
+import org.apache.spark.sql.test.SharedSparkSession
+
+import io.delta.sql.DeltaSparkSessionExtension
+
+// spotless:off
+/**
+ * A trait for tests that are testing a fully set up SparkSession with all of Delta's requirements,
+ * such as the configuration of the DeltaCatalog and the addition of all Delta extensions.
+ */
+trait DeltaSQLCommandTest extends SharedSparkSession {
+
+ override protected def sparkConf: SparkConf = {
+ val conf = super.sparkConf
+
+ // Delta.
+ conf.set(StaticSQLConf.SPARK_SESSION_EXTENSIONS.key,
+ classOf[DeltaSparkSessionExtension].getName)
+ .set(SQLConf.V2_SESSION_CATALOG_IMPLEMENTATION.key,
+ classOf[DeltaCatalog].getName)
+
+ // Gluten.
+ conf.set("spark.plugins", "org.apache.gluten.GlutenPlugin")
+ .set("spark.shuffle.manager", "org.apache.spark.shuffle.sort.ColumnarShuffleManager")
+ .set("spark.default.parallelism", "1")
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.sql.shuffle.partitions", "1")
+ .set("spark.memory.offHeap.size", "2g")
+ .set("spark.unsafe.exceptionOnMemoryLeak", "true")
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaSQLTestUtils.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaSQLTestUtils.scala
new file mode 100644
index 0000000000..22f4e9fa11
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaSQLTestUtils.scala
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta.test
+
+import org.apache.spark.sql.test.SQLTestUtils
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+// spotless:off
+trait DeltaSQLTestUtils extends SQLTestUtils {
+ /**
+ * Override the temp dir/path creation methods from [[SQLTestUtils]] to:
+ * 1. Drop the call to `waitForTasksToFinish` which is a source of flakiness due to timeouts
+ * without clear benefits.
+ * 2. Allow creating paths with special characters for better test coverage.
+ */
+
+ protected val defaultTempDirPrefix: String = "spark%dir%prefix"
+
+ override protected def withTempDir(f: File => Unit): Unit = {
+ withTempDir(prefix = defaultTempDirPrefix)(f)
+ }
+
+ override protected def withTempPaths(numPaths: Int)(f: Seq[File] => Unit): Unit = {
+ withTempPaths(numPaths, prefix = defaultTempDirPrefix)(f)
+ }
+
+ override def withTempPath(f: File => Unit): Unit = {
+ withTempPath(prefix = defaultTempDirPrefix)(f)
+ }
+
+ /**
+ * Creates a temporary directory, which is then passed to `f` and will be deleted after `f`
+ * returns.
+ */
+ def withTempDir(prefix: String)(f: File => Unit): Unit = {
+ val path = Utils.createTempDir(namePrefix = prefix)
+ try f(path) finally Utils.deleteRecursively(path)
+ }
+
+ /**
+ * Generates a temporary directory path without creating the actual directory, which is then
+ * passed to `f` and will be deleted after `f` returns.
+ */
+ def withTempPath(prefix: String)(f: File => Unit): Unit = {
+ val path = Utils.createTempDir(namePrefix = prefix)
+ path.delete()
+ try f(path) finally Utils.deleteRecursively(path)
+ }
+
+ /**
+ * Generates the specified number of temporary directory paths without creating the actual
+ * directories, which are then passed to `f` and will be deleted after `f` returns.
+ */
+ protected def withTempPaths(numPaths: Int, prefix: String)(f: Seq[File] => Unit): Unit = {
+ val files =
+ Seq.fill[File](numPaths)(Utils.createTempDir(namePrefix = prefix).getCanonicalFile)
+ files.foreach(_.delete())
+ try f(files) finally {
+ files.foreach(Utils.deleteRecursively)
+ }
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaTestImplicits.scala b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaTestImplicits.scala
new file mode 100644
index 0000000000..f2e7acc695
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/org/apache/spark/sql/delta/test/DeltaTestImplicits.scala
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.delta.test
+
+import org.apache.spark.sql.{SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
+import org.apache.spark.sql.delta.{DeltaLog, OptimisticTransaction, Snapshot}
+import org.apache.spark.sql.delta.DeltaOperations.{ManualUpdate, Operation, Write}
+import org.apache.spark.sql.delta.actions.{Action, AddFile, Metadata, Protocol}
+import org.apache.spark.sql.delta.catalog.DeltaTableV2
+import org.apache.spark.sql.delta.commands.optimize.OptimizeMetrics
+import org.apache.spark.sql.delta.coordinatedcommits.TableCommitCoordinatorClient
+import org.apache.spark.sql.delta.hooks.AutoCompact
+import org.apache.spark.sql.delta.stats.StatisticsCollection
+import org.apache.spark.util.Clock
+
+import io.delta.storage.commit.{CommitResponse, GetCommitsResponse, UpdatedActions}
+import org.apache.hadoop.fs.Path
+
+import java.io.File
+
+// spotless:off
+/**
+ * Additional method definitions for Delta classes that are intended for use only in testing.
+ */
+object DeltaTestImplicits {
+ implicit class OptimisticTxnTestHelper(txn: OptimisticTransaction) {
+
+ /** Ensure that the initial commit of a Delta table always contains a Metadata action */
+ def commitActions(op: Operation, actions: Action*): Long = {
+ if (txn.readVersion == -1) {
+ val metadataOpt = actions.collectFirst { case m: Metadata => m }
+ val protocolOpt = actions.collectFirst { case p: Protocol => p }
+ val otherActions =
+ actions.filterNot(a => a.isInstanceOf[Metadata] || a.isInstanceOf[Protocol])
+ (metadataOpt, protocolOpt) match {
+ case (Some(metadata), Some(protocol)) =>
+ // When both metadata and protocol are explicitly passed, use them.
+ txn.updateProtocol(protocol)
+ // This will auto upgrade any required table features in the passed protocol as per
+ // given metadata.
+ txn.updateMetadataForNewTable(metadata)
+ case (Some(metadata), None) =>
+ // When just metadata is passed, use it.
+ // This will auto generate protocol as per metadata.
+ txn.updateMetadataForNewTable(metadata)
+ case (None, Some(protocol)) =>
+ txn.updateProtocol(protocol)
+ txn.updateMetadataForNewTable(Metadata())
+ case (None, None) =>
+ // If neither metadata nor protocol is explicitly passed, then use default Metadata and
+ // with the maximum protocol.
+ txn.updateMetadataForNewTable(Metadata())
+ txn.updateProtocol(Action.supportedProtocolVersion())
+ }
+ txn.commit(otherActions, op)
+ } else {
+ txn.commit(actions, op)
+ }
+ }
+
+ def commitManually(actions: Action*): Long = {
+ commitActions(ManualUpdate, actions: _*)
+ }
+
+ def commitWriteAppend(actions: Action*): Long = {
+ commitActions(Write(SaveMode.Append), actions: _*)
+ }
+ }
+
+ /** Add test-only File overloads for DeltaTable.forPath */
+ implicit class DeltaLogObjectTestHelper(deltaLog: DeltaLog.type) {
+ def forTable(spark: SparkSession, dataPath: File): DeltaLog = {
+ DeltaLog.forTable(spark, new Path(dataPath.getCanonicalPath))
+ }
+
+ def forTable(spark: SparkSession, dataPath: File, clock: Clock): DeltaLog = {
+ DeltaLog.forTable(spark, new Path(dataPath.getCanonicalPath), clock)
+ }
+ }
+
+ /** Helper class for working with [[TableCommitCoordinatorClient]] */
+ implicit class TableCommitCoordinatorClientTestHelper(
+ tableCommitCoordinatorClient: TableCommitCoordinatorClient) {
+
+ def commit(
+ commitVersion: Long,
+ actions: Iterator[String],
+ updatedActions: UpdatedActions): CommitResponse = {
+ tableCommitCoordinatorClient.commit(
+ commitVersion, actions, updatedActions, tableIdentifierOpt = None)
+ }
+
+ def getCommits(
+ startVersion: Option[Long] = None,
+ endVersion: Option[Long] = None): GetCommitsResponse = {
+ tableCommitCoordinatorClient.getCommits(tableIdentifierOpt = None, startVersion, endVersion)
+ }
+
+ def backfillToVersion(
+ version: Long,
+ lastKnownBackfilledVersion: Option[Long] = None): Unit = {
+ tableCommitCoordinatorClient.backfillToVersion(
+ tableIdentifierOpt = None, version, lastKnownBackfilledVersion)
+ }
+ }
+
+
+ /** Helper class for working with [[Snapshot]] */
+ implicit class SnapshotTestHelper(snapshot: Snapshot) {
+ def ensureCommitFilesBackfilled(): Unit = {
+ snapshot.ensureCommitFilesBackfilled(catalogTableOpt = None)
+ }
+ }
+
+ /**
+ * Helper class for working with the most recent snapshot in the deltaLog
+ */
+ implicit class DeltaLogTestHelper(deltaLog: DeltaLog) {
+ def snapshot: Snapshot = {
+ deltaLog.unsafeVolatileSnapshot
+ }
+
+ def checkpoint(): Unit = {
+ deltaLog.checkpoint(snapshot)
+ }
+
+ def checkpointInterval(): Int = {
+ deltaLog.checkpointInterval(snapshot.metadata)
+ }
+
+ def deltaRetentionMillis(): Long = {
+ deltaLog.deltaRetentionMillis(snapshot.metadata)
+ }
+
+ def enableExpiredLogCleanup(): Boolean = {
+ deltaLog.enableExpiredLogCleanup(snapshot.metadata)
+ }
+
+ def upgradeProtocol(newVersion: Protocol): Unit = {
+ upgradeProtocol(deltaLog.unsafeVolatileSnapshot, newVersion)
+ }
+
+ def upgradeProtocol(snapshot: Snapshot, newVersion: Protocol): Unit = {
+ deltaLog.upgradeProtocol(None, snapshot, newVersion)
+ }
+ }
+
+ implicit class DeltaTableV2ObjectTestHelper(dt: DeltaTableV2.type) {
+ /** Convenience overload that omits the cmd arg (which is not helpful in tests). */
+ def apply(spark: SparkSession, id: TableIdentifier): DeltaTableV2 =
+ dt.apply(spark, id, "test")
+ }
+
+ implicit class DeltaTableV2TestHelper(deltaTable: DeltaTableV2) {
+ /** For backward compatibility with existing unit tests */
+ def snapshot: Snapshot = deltaTable.initialSnapshot
+ }
+
+ implicit class AutoCompactObjectTestHelper(ac: AutoCompact.type) {
+ private[delta] def compact(
+ spark: SparkSession,
+ deltaLog: DeltaLog,
+ partitionPredicates: Seq[Expression] = Nil,
+ opType: String = AutoCompact.OP_TYPE): Seq[OptimizeMetrics] = {
+ AutoCompact.compact(
+ spark, deltaLog, catalogTable = None,
+ partitionPredicates, opType)
+ }
+ }
+
+ implicit class StatisticsCollectionObjectTestHelper(sc: StatisticsCollection.type) {
+
+ /**
+ * This is an implicit helper required for backward compatibility with existing
+ * unit tests. It allows to call [[StatisticsCollection.recompute]] without a
+ * catalog table and in the actual call, sets it to [[None]].
+ */
+ def recompute(
+ spark: SparkSession,
+ deltaLog: DeltaLog,
+ predicates: Seq[Expression] = Seq(Literal(true)),
+ fileFilter: AddFile => Boolean = af => true): Unit = {
+ StatisticsCollection.recompute(
+ spark, deltaLog, catalogTable = None, predicates, fileFilter)
+ }
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-delta33/test/scala/shims/DeltaExcludedBySparkVersionTestMixinShims.scala b/backends-bolt/src-delta33/test/scala/shims/DeltaExcludedBySparkVersionTestMixinShims.scala
new file mode 100644
index 0000000000..26c1a69481
--- /dev/null
+++ b/backends-bolt/src-delta33/test/scala/shims/DeltaExcludedBySparkVersionTestMixinShims.scala
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package shims
+
+import org.apache.spark.sql.QueryTest
+
+// spotless:off
+trait DeltaExcludedBySparkVersionTestMixinShims extends QueryTest {
+ /**
+ * Tests that are meant for Delta compiled against Spark Latest Release only. Executed since this
+ * is the Spark Latest Release shim.
+ */
+ protected def testSparkLatestOnly(
+ testName: String, testTags: org.scalatest.Tag*)
+ (testFun: => Any)
+ (implicit pos: org.scalactic.source.Position): Unit = {
+ test(testName, testTags: _*)(testFun)(pos)
+ }
+
+ /**
+ * Tests that are meant for Delta compiled against Spark Master Release only. Ignored since this
+ * is the Spark Latest Release shim.
+ */
+ protected def testSparkMasterOnly(
+ testName: String, testTags: org.scalatest.Tag*)
+ (testFun: => Any)
+ (implicit pos: org.scalactic.source.Position): Unit = {
+ ignore(testName, testTags: _*)(testFun)(pos)
+ }
+}
+// spotless:on
diff --git a/backends-bolt/src-hudi/main/resources/META-INF/gluten-components/org.apache.gluten.component.BoltHudiComponent b/backends-bolt/src-hudi/main/resources/META-INF/gluten-components/org.apache.gluten.component.BoltHudiComponent
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/backends-bolt/src-hudi/main/scala/org/apache/gluten/component/BoltHudiComponent.scala b/backends-bolt/src-hudi/main/scala/org/apache/gluten/component/BoltHudiComponent.scala
new file mode 100644
index 0000000000..59060b79f3
--- /dev/null
+++ b/backends-bolt/src-hudi/main/scala/org/apache/gluten/component/BoltHudiComponent.scala
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.component
+
+import org.apache.gluten.backendsapi.bolt.BoltBackend
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.OffloadHudiScan
+import org.apache.gluten.extension.columnar.enumerated.RasOffload
+import org.apache.gluten.extension.columnar.heuristic.HeuristicTransform
+import org.apache.gluten.extension.columnar.validator.Validators
+import org.apache.gluten.extension.injector.Injector
+
+import org.apache.spark.sql.execution.FileSourceScanExec
+
+class BoltHudiComponent extends Component {
+ override def name(): String = "bolt-hudi"
+ override def buildInfo(): Component.BuildInfo =
+ Component.BuildInfo("BoltHudi", "N/A", "N/A", "N/A")
+ override def dependencies(): Seq[Class[_ <: Component]] = classOf[BoltBackend] :: Nil
+ override def injectRules(injector: Injector): Unit = {
+ val legacy = injector.gluten.legacy
+ val ras = injector.gluten.ras
+ legacy.injectTransform {
+ c =>
+ val offload = Seq(OffloadHudiScan()).map(_.toStrcitRule())
+ HeuristicTransform.Simple(
+ Validators.newValidator(new GlutenConfig(c.sqlConf), offload),
+ offload)
+ }
+ ras.injectRasRule {
+ c =>
+ RasOffload.Rule(
+ RasOffload.from[FileSourceScanExec](OffloadHudiScan()),
+ Validators.newValidator(new GlutenConfig(c.sqlConf)),
+ Nil)
+ }
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-hudi/test/scala/org/apache/gluten/execution/BoltHudiSuite.scala
similarity index 81%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-hudi/test/scala/org/apache/gluten/execution/BoltHudiSuite.scala
index 632c806075..05d117a4ea 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-hudi/test/scala/org/apache/gluten/execution/BoltHudiSuite.scala
@@ -14,13 +14,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.execution
-#pragma once
-
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+class BoltHudiSuite extends HudiSuite {}
diff --git a/backends-bolt/src-hudi/test/scala/org/apache/gluten/execution/BoltTPCHHudiSuite.scala b/backends-bolt/src-hudi/test/scala/org/apache/gluten/execution/BoltTPCHHudiSuite.scala
new file mode 100644
index 0000000000..74b6e2fc8c
--- /dev/null
+++ b/backends-bolt/src-hudi/test/scala/org/apache/gluten/execution/BoltTPCHHudiSuite.scala
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.spark.SparkConf
+
+import java.io.File
+
+class BoltTPCHHudiSuite extends BoltTPCHSuite {
+ protected val tpchBasePath: String =
+ getClass.getResource("/").getPath + "../../../src/test/resources"
+
+ override protected val resourcePath: String =
+ getClass.getResource("/tpch-data-parquet").getFile
+
+ override protected val queriesResults: String =
+ new File(tpchBasePath, "queries-output").getCanonicalPath
+
+ override protected def sparkConf: SparkConf = {
+ super.sparkConf
+ .set("spark.executor.memory", "4g")
+ .set("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
+ .set("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ .set("spark.kryo.registrator", "org.apache.spark.HoodieSparkKryoRegistrar")
+ .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ }
+
+ override protected def createTPCHNotNullTables(): Unit = {
+ val tableDir = getClass.getResource(resourcePath).getFile
+ TPCHTables
+ .map(_.name)
+ .map {
+ table =>
+ val tablePath = new File(tableDir, table).getAbsolutePath
+ val tableDF = spark.read.format(fileFormat).load(tablePath)
+ tableDF.write.format("hudi").mode("append").saveAsTable(table)
+ (table, tableDF)
+ }
+ .toMap
+ }
+
+ override protected def afterAll(): Unit = {
+ TPCHTables.map(_.name).foreach(table => spark.sql(s"DROP TABLE IF EXISTS $table"))
+ super.afterAll()
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestStoragePartitionedJoins.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestStoragePartitionedJoins.java
new file mode 100644
index 0000000000..8123ca16fc
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestStoragePartitionedJoins.java
@@ -0,0 +1,665 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.spark.SparkSchemaUtil;
+import org.apache.iceberg.spark.SparkTestBaseWithCatalog;
+import org.apache.iceberg.spark.SparkWriteOptions;
+import org.apache.iceberg.spark.data.RandomData;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.internal.SQLConf;
+import org.apache.spark.sql.types.StructType;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.iceberg.PlanningMode.DISTRIBUTED;
+import static org.apache.iceberg.PlanningMode.LOCAL;
+
+@RunWith(Parameterized.class)
+public class TestStoragePartitionedJoins extends SparkTestBaseWithCatalog {
+
+ @Parameterized.Parameters(name = "planningMode = {0}")
+ public static Object[] parameters() {
+ return new Object[] {LOCAL, DISTRIBUTED};
+ }
+
+ private static final String OTHER_TABLE_NAME = "other_table";
+
+ // open file cost and split size are set as 16 MB to produce a split per file
+ private static final Map<String, String> TABLE_PROPERTIES =
+ ImmutableMap.of(
+ TableProperties.SPLIT_SIZE, "16777216", TableProperties.SPLIT_OPEN_FILE_COST, "16777216");
+
+ // only v2 bucketing and preserve data grouping properties have to be enabled to trigger SPJ
+ // other properties are only to simplify testing and validation
+ private static final Map<String, String> ENABLED_SPJ_SQL_CONF =
+ ImmutableMap.of(
+ SQLConf.V2_BUCKETING_ENABLED().key(),
+ "true",
+ SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED().key(),
+ "true",
+ SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION().key(),
+ "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(),
+ "false",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD().key(),
+ "-1",
+ SparkSQLProperties.PRESERVE_DATA_GROUPING,
+ "true");
+
+ private static final Map<String, String> DISABLED_SPJ_SQL_CONF =
+ ImmutableMap.of(
+ SQLConf.V2_BUCKETING_ENABLED().key(),
+ "false",
+ SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION().key(),
+ "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(),
+ "false",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD().key(),
+ "-1",
+ SparkSQLProperties.PRESERVE_DATA_GROUPING,
+ "true");
+
+ private final PlanningMode planningMode;
+
+ public TestStoragePartitionedJoins(PlanningMode planningMode) {
+ this.planningMode = planningMode;
+ }
+
+ @BeforeClass
+ public static void setupSparkConf() {
+ spark.conf().set("spark.sql.shuffle.partitions", "4");
+ }
+
+ @After
+ public void removeTables() {
+ sql("DROP TABLE IF EXISTS %s", tableName);
+ sql("DROP TABLE IF EXISTS %s", tableName(OTHER_TABLE_NAME));
+ }
+
+ // TODO: add tests for truncate transforms once SPARK-40295 is released
+
+ @Test
+ public void testJoinsWithBucketingOnByteColumn() throws NoSuchTableException {
+ checkJoin("byte_col", "TINYINT", "bucket(4, byte_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnShortColumn() throws NoSuchTableException {
+ checkJoin("short_col", "SMALLINT", "bucket(4, short_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnIntColumn() throws NoSuchTableException {
+ checkJoin("int_col", "INT", "bucket(16, int_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnLongColumn() throws NoSuchTableException {
+ checkJoin("long_col", "BIGINT", "bucket(16, long_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnTimestampColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP", "bucket(16, timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnTimestampNtzColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP_NTZ", "bucket(16, timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnDateColumn() throws NoSuchTableException {
+ checkJoin("date_col", "DATE", "bucket(8, date_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnDecimalColumn() throws NoSuchTableException {
+ checkJoin("decimal_col", "DECIMAL(20, 2)", "bucket(8, decimal_col)");
+ }
+
+ @Test
+ public void testJoinsWithBucketingOnBinaryColumn() throws NoSuchTableException {
+ checkJoin("binary_col", "BINARY", "bucket(8, binary_col)");
+ }
+
+ @Test
+ public void testJoinsWithYearsOnTimestampColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP", "years(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithYearsOnTimestampNtzColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP_NTZ", "years(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithYearsOnDateColumn() throws NoSuchTableException {
+ checkJoin("date_col", "DATE", "years(date_col)");
+ }
+
+ @Test
+ public void testJoinsWithMonthsOnTimestampColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP", "months(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithMonthsOnTimestampNtzColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP_NTZ", "months(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithMonthsOnDateColumn() throws NoSuchTableException {
+ checkJoin("date_col", "DATE", "months(date_col)");
+ }
+
+ @Test
+ public void testJoinsWithDaysOnTimestampColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP", "days(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithDaysOnTimestampNtzColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP_NTZ", "days(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithDaysOnDateColumn() throws NoSuchTableException {
+ checkJoin("date_col", "DATE", "days(date_col)");
+ }
+
+ @Test
+ public void testJoinsWithHoursOnTimestampColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP", "hours(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithHoursOnTimestampNtzColumn() throws NoSuchTableException {
+ checkJoin("timestamp_col", "TIMESTAMP_NTZ", "hours(timestamp_col)");
+ }
+
+ @Test
+ public void testJoinsWithMultipleTransformTypes() throws NoSuchTableException {
+ String createTableStmt =
+ "CREATE TABLE %s ("
+ + " id BIGINT, int_col INT, date_col1 DATE, date_col2 DATE, date_col3 DATE,"
+ + " timestamp_col TIMESTAMP, string_col STRING, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY ("
+ + " years(date_col1), months(date_col2), days(date_col3), hours(timestamp_col), "
+ + " bucket(8, int_col), dep)"
+ + "TBLPROPERTIES (%s)";
+
+ sql(createTableStmt, tableName, tablePropsAsString(TABLE_PROPERTIES));
+ sql(createTableStmt, tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ Table table = validationCatalog.loadTable(tableIdent);
+
+ Dataset<Row> dataDF = randomDataDF(table.schema(), 16);
+
+ // write to the first table 1 time to generate 1 file per partition
+ append(tableName, dataDF);
+
+ // write to the second table 2 times to generate 2 files per partition
+ append(tableName(OTHER_TABLE_NAME), dataDF);
+ append(tableName(OTHER_TABLE_NAME), dataDF);
+
+ // Spark SPJ support is limited at the moment and requires all source partitioning columns,
+ // which were projected in the query, to be part of the join condition
+ // suppose a table is partitioned by `p1`, `bucket(8, pk)`
+ // queries covering `p1` and `pk` columns must include equality predicates
+ // on both `p1` and `pk` to benefit from SPJ
+ // this is a temporary Spark limitation that will be removed in a future release
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT t1.id "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.dep = t2.dep "
+ + "ORDER BY t1.id",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT t1.id, t1.int_col, t1.date_col1 "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.int_col = t2.int_col AND t1.date_col1 = t2.date_col1 "
+ + "ORDER BY t1.id, t1.int_col, t1.date_col1",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT t1.id, t1.timestamp_col, t1.string_col "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.timestamp_col = t2.timestamp_col AND t1.string_col = t2.string_col "
+ + "ORDER BY t1.id, t1.timestamp_col, t1.string_col",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT t1.id, t1.date_col1, t1.date_col2, t1.date_col3 "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.date_col1 = t2.date_col1 AND t1.date_col2 = t2.date_col2 AND t1.date_col3 = t2.date_col3 "
+ + "ORDER BY t1.id, t1.date_col1, t1.date_col2, t1.date_col3",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT t1.id, t1.int_col, t1.timestamp_col, t1.dep "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.int_col = t2.int_col AND t1.timestamp_col = t2.timestamp_col AND t1.dep = t2.dep "
+ + "ORDER BY t1.id, t1.int_col, t1.timestamp_col, t1.dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testJoinsWithCompatibleSpecEvolution() {
+ // create a table with an empty spec
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ Table table = validationCatalog.loadTable(tableIdent);
+
+ // evolve the spec in the first table by adding `dep`
+ table.updateSpec().addField("dep").commit();
+
+ // insert data into the first table partitioned by `dep`
+ sql("REFRESH TABLE %s", tableName);
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName);
+
+ // evolve the spec in the first table by adding `bucket(int_col, 8)`
+ table.updateSpec().addField(Expressions.bucket("int_col", 8)).commit();
+
+ // insert data into the first table partitioned by `dep`, `bucket(8, int_col)`
+ sql("REFRESH TABLE %s", tableName);
+ sql("INSERT INTO %s VALUES (2L, 200, 'hr')", tableName);
+
+ // create another table partitioned by `other_dep`
+ sql(
+ "CREATE TABLE %s (other_id BIGINT, other_int_col INT, other_dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (other_dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ // insert data into the second table partitioned by 'other_dep'
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (2L, 200, 'hr')", tableName(OTHER_TABLE_NAME));
+
+ // SPJ would apply as the grouping keys are compatible
+ // the first table: `dep` (an intersection of all active partition fields across scanned specs)
+ // the second table: `other_dep` (the only partition field).
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT * "
+ + "FROM %s "
+ + "INNER JOIN %s "
+ + "ON id = other_id AND int_col = other_int_col AND dep = other_dep "
+ + "ORDER BY id, int_col, dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testJoinsWithIncompatibleSpecs() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')", tableName);
+
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (bucket(8, int_col))"
+ + "TBLPROPERTIES (%s)",
+ tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')", tableName(OTHER_TABLE_NAME));
+
+ // queries can't benefit from SPJ as specs are not compatible
+ // the first table: `dep`
+ // the second table: `bucket(8, int_col)`
+
+ assertPartitioningAwarePlan(
+ 3, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles with SPJ */
+ "SELECT * "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.int_col = t2.int_col AND t1.dep = t2.dep "
+ + "ORDER BY t1.id, t1.int_col, t1.dep, t2.id, t2.int_col, t2.dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testJoinsWithUnpartitionedTables() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "TBLPROPERTIES ("
+ + " 'read.split.target-size' = 16777216,"
+ + " 'read.split.open-file-cost' = 16777216)",
+ tableName);
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')", tableName);
+
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "TBLPROPERTIES ("
+ + " 'read.split.target-size' = 16777216,"
+ + " 'read.split.open-file-cost' = 16777216)",
+ tableName(OTHER_TABLE_NAME));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')", tableName(OTHER_TABLE_NAME));
+
+ // queries covering unpartitioned tables can't benefit from SPJ but shouldn't fail
+
+ assertPartitioningAwarePlan(
+ 3, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT * "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.int_col = t2.int_col AND t1.dep = t2.dep "
+ + "ORDER BY t1.id, t1.int_col, t1.dep, t2.id, t2.int_col, t2.dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testJoinsWithEmptyTable() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (2L, 200, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (3L, 300, 'software')", tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 3, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT * "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.int_col = t2.int_col AND t1.dep = t2.dep "
+ + "ORDER BY t1.id, t1.int_col, t1.dep, t2.id, t2.int_col, t2.dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testJoinsWithOneSplitTables() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName);
+
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName(OTHER_TABLE_NAME));
+
+ // Spark should be able to avoid shuffles even without SPJ if each side has only one split
+
+ assertPartitioningAwarePlan(
+ 0, /* expected num of shuffles with SPJ */
+ 0, /* expected num of shuffles without SPJ */
+ "SELECT * "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.int_col = t2.int_col AND t1.dep = t2.dep "
+ + "ORDER BY t1.id, t1.int_col, t1.dep, t2.id, t2.int_col, t2.dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testJoinsWithMismatchingPartitionKeys() {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName);
+ sql("INSERT INTO %s VALUES (2L, 100, 'hr')", tableName);
+
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep)"
+ + "TBLPROPERTIES (%s)",
+ tableName(OTHER_TABLE_NAME), tablePropsAsString(TABLE_PROPERTIES));
+
+ sql("INSERT INTO %s VALUES (1L, 100, 'software')", tableName(OTHER_TABLE_NAME));
+ sql("INSERT INTO %s VALUES (3L, 300, 'hardware')", tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT * "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.dep = t2.dep "
+ + "ORDER BY t1.id, t1.int_col, t1.dep, t2.id, t2.int_col, t2.dep",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ @Test
+ public void testAggregates() throws NoSuchTableException {
+ sql(
+ "CREATE TABLE %s (id BIGINT, int_col INT, dep STRING)"
+ + "USING iceberg "
+ + "PARTITIONED BY (dep, bucket(8, int_col))"
+ + "TBLPROPERTIES (%s)",
+ tableName, tablePropsAsString(TABLE_PROPERTIES));
+
+ // write to the table 3 times to generate 3 files per partition
+ Table table = validationCatalog.loadTable(tableIdent);
+ Dataset<Row> dataDF = randomDataDF(table.schema(), 100);
+ append(tableName, dataDF);
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT COUNT (DISTINCT id) AS count FROM %s GROUP BY dep, int_col ORDER BY count",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT COUNT (DISTINCT id) AS count FROM %s GROUP BY dep ORDER BY count",
+ tableName,
+ tableName(OTHER_TABLE_NAME));
+ }
+
+ private void checkJoin(String sourceColumnName, String sourceColumnType, String transform)
+ throws NoSuchTableException {
+
+ String createTableStmt =
+ "CREATE TABLE %s (id BIGINT, salary INT, %s %s)"
+ + "USING iceberg "
+ + "PARTITIONED BY (%s)"
+ + "TBLPROPERTIES (%s)";
+
+ sql(
+ createTableStmt,
+ tableName,
+ sourceColumnName,
+ sourceColumnType,
+ transform,
+ tablePropsAsString(TABLE_PROPERTIES));
+ configurePlanningMode(tableName, planningMode);
+
+ sql(
+ createTableStmt,
+ tableName(OTHER_TABLE_NAME),
+ sourceColumnName,
+ sourceColumnType,
+ transform,
+ tablePropsAsString(TABLE_PROPERTIES));
+ configurePlanningMode(tableName(OTHER_TABLE_NAME), planningMode);
+
+ Table table = validationCatalog.loadTable(tableIdent);
+ Dataset<Row> dataDF = randomDataDF(table.schema(), 200);
+ append(tableName, dataDF);
+ append(tableName(OTHER_TABLE_NAME), dataDF);
+
+ assertPartitioningAwarePlan(
+ 1, /* expected num of shuffles with SPJ */
+ 3, /* expected num of shuffles without SPJ */
+ "SELECT t1.id, t1.salary, t1.%s "
+ + "FROM %s t1 "
+ + "INNER JOIN %s t2 "
+ + "ON t1.id = t2.id AND t1.%s = t2.%s "
+ + "ORDER BY t1.id, t1.%s, t1.salary", // add order by salary to make test stable
+ sourceColumnName,
+ tableName,
+ tableName(OTHER_TABLE_NAME),
+ sourceColumnName,
+ sourceColumnName,
+ sourceColumnName);
+ }
+
+ private void assertPartitioningAwarePlan(
+ int expectedNumShufflesWithSPJ,
+ int expectedNumShufflesWithoutSPJ,
+ String query,
+ Object... args) {
+
+ AtomicReference<List<Object[]>> rowsWithSPJ = new AtomicReference<>();
+ AtomicReference<List<Object[]>> rowsWithoutSPJ = new AtomicReference<>();
+
+ withSQLConf(
+ ENABLED_SPJ_SQL_CONF,
+ () -> {
+ String plan = executeAndKeepPlan(query, args).toString();
+ int actualNumShuffles = StringUtils.countMatches(plan, "Exchange");
+ Assert.assertEquals(
+ "Number of shuffles with enabled SPJ must match",
+ expectedNumShufflesWithSPJ,
+ actualNumShuffles);
+
+ rowsWithSPJ.set(sql(query, args));
+ });
+
+ withSQLConf(
+ DISABLED_SPJ_SQL_CONF,
+ () -> {
+ String plan = executeAndKeepPlan(query, args).toString();
+ int actualNumShuffles = StringUtils.countMatches(plan, "Exchange");
+ Assert.assertEquals(
+ "Number of shuffles with disabled SPJ must match",
+ expectedNumShufflesWithoutSPJ,
+ actualNumShuffles);
+
+ rowsWithoutSPJ.set(sql(query, args));
+ });
+
+ assertEquals("SPJ should not change query output", rowsWithoutSPJ.get(), rowsWithSPJ.get());
+ }
+
+ private Dataset<Row> randomDataDF(Schema schema, int numRows) {
+ Iterable<InternalRow> rows = RandomData.generateSpark(schema, numRows, 0);
+ JavaRDD<InternalRow> rowRDD = sparkContext.parallelize(Lists.newArrayList(rows));
+ StructType rowSparkType = SparkSchemaUtil.convert(schema);
+ return spark.internalCreateDataFrame(JavaRDD.toRDD(rowRDD), rowSparkType, false);
+ }
+
+ private void append(String table, Dataset<Row> df) throws NoSuchTableException {
+ // fanout writes are enabled as write-time clustering is not supported without Spark extensions
+ df.coalesce(1).writeTo(table).option(SparkWriteOptions.FANOUT_ENABLED, "true").append();
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java
new file mode 100644
index 0000000000..0d7379963d
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/execution/TestTPCHStoragePartitionedJoins.java
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution;
+
+import org.apache.gluten.config.GlutenConfig;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.spark.SparkTestBaseWithCatalog;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.internal.SQLConf;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class TestTPCHStoragePartitionedJoins extends SparkTestBaseWithCatalog {
+ protected String rootPath = this.getClass().getResource("/").getPath();
+
+ protected String tpchQueries =
+ rootPath + "../../../../tools/gluten-it/common/src/main/resources/tpch-queries";
+
+ // open file cost and split size are set as 16 MB to produce a split per file
+ private static final Map<String, String> TABLE_PROPERTIES =
+ ImmutableMap.of(
+ TableProperties.SPLIT_SIZE, "16777216", TableProperties.SPLIT_OPEN_FILE_COST, "16777216");
+
+ // only v2 bucketing and preserve data grouping properties have to be enabled to trigger SPJ
+ // other properties are only to simplify testing and validation
+ private static final Map<String, String> ENABLED_SPJ_SQL_CONF =
+ ImmutableMap.of(
+ SQLConf.V2_BUCKETING_ENABLED().key(),
+ "true",
+ SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED().key(),
+ "true",
+ SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION().key(),
+ "false",
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(),
+ "false",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD().key(),
+ "-1",
+ SparkSQLProperties.PRESERVE_DATA_GROUPING,
+ "true",
+ SQLConf.V2_BUCKETING_PARTIALLY_CLUSTERED_DISTRIBUTION_ENABLED().key(),
+ "true");
+ protected static String PARQUET_TABLE_PREFIX = "p_";
+ protected static List<String> tableNames =
+ ImmutableList.of(
+ "part", "supplier", "partsupp", "customer", "orders", "lineitem", "nation", "region");
+
+ // If we test all the catalog, we need to create the table in that catalog,
+ // we don't need to test the catalog, so only test the testhadoop catalog
+ @Before
+ public void createTPCHNotNullTables() {
+ final String tableDir = this.getClass().getResource("/tpch-data-parquet").getFile();
+ tableNames.forEach(
+ table -> {
+ String tablePath = new File(tableDir, table).getAbsolutePath();
+ Dataset<Row> tableDF = spark.read().format("parquet").load(tablePath);
+ tableDF.createOrReplaceTempView(PARQUET_TABLE_PREFIX + table);
+ });
+
+ sql(
+ createIcebergTable(
+ "part",
+ "`p_partkey` INT,\n"
+ + " `p_name` string,\n"
+ + " `p_mfgr` string,\n"
+ + " `p_brand` string,\n"
+ + " `p_type` string,\n"
+ + " `p_size` INT,\n"
+ + " `p_container` string,\n"
+ + " `p_retailprice` DECIMAL(15,2) ,\n"
+ + " `p_comment` string ",
+ null));
+ sql(
+ createIcebergTable(
+ "nation",
+ "`n_nationkey` INT,\n"
+ + " `n_name` CHAR(25),\n"
+ + " `n_regionkey` INT,\n"
+ + " `n_comment` VARCHAR(152)"));
+ sql(
+ createIcebergTable(
+ "region",
+ "`r_regionkey` INT,\n"
+ + " `r_name` CHAR(25),\n"
+ + " `r_comment` VARCHAR(152)"));
+ sql(
+ createIcebergTable(
+ "supplier",
+ "`s_suppkey` INT,\n"
+ + " `s_name` CHAR(25),\n"
+ + " `s_address` VARCHAR(40),\n"
+ + " `s_nationkey` INT,\n"
+ + " `s_phone` CHAR(15),\n"
+ + " `s_acctbal` DECIMAL(15,2),\n"
+ + " `s_comment` VARCHAR(101)"));
+ sql(
+ createIcebergTable(
+ "customer",
+ "`c_custkey` INT,\n"
+ + " `c_name` string,\n"
+ + " `c_address` string,\n"
+ + " `c_nationkey` INT,\n"
+ + " `c_phone` string,\n"
+ + " `c_acctbal` DECIMAL(15,2),\n"
+ + " `c_mktsegment` string,\n"
+ + " `c_comment` string",
+ "bucket(16, c_custkey)"));
+ sql(
+ createIcebergTable(
+ "partsupp",
+ "`ps_partkey` INT,\n"
+ + " `ps_suppkey` INT,\n"
+ + " `ps_availqty` INT,\n"
+ + " `ps_supplycost` DECIMAL(15,2),\n"
+ + " `ps_comment` VARCHAR(199)"));
+ sql(
+ createIcebergTable(
+ "orders",
+ "`o_orderkey` INT,\n"
+ + " `o_custkey` INT,\n"
+ + " `o_orderstatus` string,\n"
+ + " `o_totalprice` DECIMAL(15,2),\n"
+ + " `o_orderdate` DATE,\n"
+ + " `o_orderpriority` string,\n"
+ + " `o_clerk` string,\n"
+ + " `o_shippriority` INT,\n"
+ + " `o_comment` string",
+ "bucket(16, o_custkey)"));
+
+ sql(
+ createIcebergTable(
+ "lineitem",
+ "`l_orderkey` INT,\n"
+ + " `l_partkey` INT,\n"
+ + " `l_suppkey` INT,\n"
+ + " `l_linenumber` INT,\n"
+ + " `l_quantity` DECIMAL(15,2),\n"
+ + " `l_extendedprice` DECIMAL(15,2),\n"
+ + " `l_discount` DECIMAL(15,2),\n"
+ + " `l_tax` DECIMAL(15,2),\n"
+ + " `l_returnflag` string,\n"
+ + " `l_linestatus` string,\n"
+ + " `l_shipdate` DATE,\n"
+ + " `l_commitdate` DATE,\n"
+ + " `l_receiptdate` DATE,\n"
+ + " `l_shipinstruct` string,\n"
+ + " `l_shipmode` string,\n"
+ + " `l_comment` string",
+ null));
+
+ String insertStmt = "INSERT INTO %s select * from %s%s";
+ tableNames.forEach(
+ table -> sql(String.format(insertStmt, tableName(table), PARQUET_TABLE_PREFIX, table)));
+ }
+
+ @After
+ public void dropTPCHNotNullTables() {
+ tableNames.forEach(
+ table -> {
+ sql("DROP TABLE IF EXISTS " + tableName(table));
+ sql("DROP VIEW IF EXISTS " + PARQUET_TABLE_PREFIX + table);
+ });
+ }
+
+ private String createIcebergTable(String name, String columns) {
+ return createIcebergTable(name, columns, null);
+ }
+
+ private String createIcebergTable(String name, String columns, String transform) {
+ // create TPCH iceberg table
+ String createTableStmt =
+ "CREATE TABLE %s (%s)" + "USING iceberg " + "PARTITIONED BY (%s)" + "TBLPROPERTIES (%s)";
+ String createUnpartitionTableStmt =
+ "CREATE TABLE %s (%s)" + "USING iceberg " + "TBLPROPERTIES (%s)";
+ if (transform != null) {
+ return String.format(
+ createTableStmt,
+ tableName(name),
+ columns,
+ transform,
+ tablePropsAsString(TABLE_PROPERTIES));
+ } else {
+ return String.format(
+ createUnpartitionTableStmt,
+ tableName(name),
+ columns,
+ tablePropsAsString(TABLE_PROPERTIES));
+ }
+ }
+
+ protected String tpchSQL(int queryNum) {
+ try {
+ return FileUtils.readFileToString(new File(tpchQueries + "/q" + queryNum + ".sql"), "UTF-8");
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void testTPCH() {
+ spark.conf().set("spark.sql.defaultCatalog", catalogName);
+ spark.conf().set("spark.sql.catalog." + catalogName + ".default-namespace", "default");
+ sql("use namespace default");
+ withSQLConf(
+ ENABLED_SPJ_SQL_CONF,
+ () -> {
+ for (int i = 1; i <= 22; i++) {
+ List<Row> rows = spark.sql(tpchSQL(i)).collectAsList();
+ AtomicReference<List<Row>> rowsSpark = new AtomicReference<>();
+ int finalI = i;
+ withSQLConf(
+ ImmutableMap.of(GlutenConfig.GLUTEN_ENABLED().key(), "false"),
+ () -> rowsSpark.set(spark.sql(tpchSQL(finalI)).collectAsList()));
+ assertThat(rows).containsExactlyInAnyOrderElementsOf(Iterables.concat(rowsSpark.get()));
+ }
+ });
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenCopyOnWriteDelete.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenCopyOnWriteDelete.java
new file mode 100644
index 0000000000..e03d4aba8c
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenCopyOnWriteDelete.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extensions;
+
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.spark.extensions.TestCopyOnWriteDelete;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+public class TestGlutenCopyOnWriteDelete extends TestCopyOnWriteDelete {
+ public TestGlutenCopyOnWriteDelete(
+ String catalogName,
+ String implementation,
+ Map<String, String> config,
+ String fileFormat,
+ Boolean vectorized,
+ String distributionMode,
+ boolean fanoutEnabled,
+ String branch,
+ PlanningMode planningMode) {
+ super(
+ catalogName,
+ implementation,
+ config,
+ fileFormat,
+ vectorized,
+ distributionMode,
+ fanoutEnabled,
+ branch,
+ planningMode);
+ }
+
+ @Test
+ public synchronized void testDeleteWithConcurrentTableRefresh() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testDeleteWithSerializableIsolation() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testDeleteWithSnapshotIsolation() throws ExecutionException {
+ System.out.println("Run timeout");
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadDelete.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadDelete.java
new file mode 100644
index 0000000000..f2fe3e3341
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadDelete.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extensions;
+
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.spark.extensions.TestMergeOnReadDelete;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+public class TestGlutenMergeOnReadDelete extends TestMergeOnReadDelete {
+ public TestGlutenMergeOnReadDelete(
+ String catalogName,
+ String implementation,
+ Map<String, String> config,
+ String fileFormat,
+ Boolean vectorized,
+ String distributionMode,
+ boolean fanoutEnabled,
+ String branch,
+ PlanningMode planningMode) {
+ super(
+ catalogName,
+ implementation,
+ config,
+ fileFormat,
+ vectorized,
+ distributionMode,
+ fanoutEnabled,
+ branch,
+ planningMode);
+ }
+
+ @Test
+ public synchronized void testDeleteWithConcurrentTableRefresh() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testDeleteWithSerializableIsolation() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testDeleteWithSnapshotIsolation() throws ExecutionException {
+ System.out.println("Run timeout");
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadMerge.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadMerge.java
new file mode 100644
index 0000000000..efb919f1b4
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadMerge.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extensions;
+
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.RowLevelOperationMode;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.extensions.TestMergeOnReadMerge;
+import org.apache.spark.sql.execution.SparkPlan;
+import org.apache.spark.sql.internal.SQLConf;
+import org.junit.Test;
+
+import java.util.Map;
+
+import static org.apache.iceberg.RowLevelOperationMode.COPY_ON_WRITE;
+import static org.apache.iceberg.TableProperties.MERGE_MODE;
+import static org.apache.iceberg.TableProperties.MERGE_MODE_DEFAULT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+public class TestGlutenMergeOnReadMerge extends TestMergeOnReadMerge {
+ public TestGlutenMergeOnReadMerge(
+ String catalogName,
+ String implementation,
+ Map<String, String> config,
+ String fileFormat,
+ boolean vectorized,
+ String distributionMode,
+ boolean fanoutEnabled,
+ String branch,
+ PlanningMode planningMode) {
+ super(
+ catalogName,
+ implementation,
+ config,
+ fileFormat,
+ vectorized,
+ distributionMode,
+ fanoutEnabled,
+ branch,
+ planningMode);
+ }
+
+ @Test
+ public synchronized void testMergeWithConcurrentTableRefresh() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testMergeWithSerializableIsolation() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testMergeWithSnapshotIsolation() {
+ System.out.println("Run timeout");
+ }
+
+ // The matched join string is changed from Join to ShuffledHashJoinExecTransformer
+ @Test
+ public void testMergeConditionSplitIntoTargetPredicateAndJoinCondition() {
+ createAndInitTable(
+ "id INT, salary INT, dep STRING, sub_dep STRING",
+ "PARTITIONED BY (dep, sub_dep)",
+ "{ \"id\": 1, \"salary\": 100, \"dep\": \"d1\", \"sub_dep\": \"sd1\" }\n"
+ + "{ \"id\": 6, \"salary\": 600, \"dep\": \"d6\", \"sub_dep\": \"sd6\" }");
+
+ createOrReplaceView(
+ "source",
+ "id INT, salary INT, dep STRING, sub_dep STRING",
+ "{ \"id\": 1, \"salary\": 101, \"dep\": \"d1\", \"sub_dep\": \"sd1\" }\n"
+ + "{ \"id\": 2, \"salary\": 200, \"dep\": \"d2\", \"sub_dep\": \"sd2\" }\n"
+ + "{ \"id\": 3, \"salary\": 300, \"dep\": \"d3\", \"sub_dep\": \"sd3\" }");
+
+ String query =
+ String.format(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id == s.id AND ((t.dep = 'd1' AND t.sub_dep IN ('sd1', 'sd3')) OR (t.dep = 'd6' AND t.sub_dep IN ('sd2', 'sd6'))) "
+ + "WHEN MATCHED THEN "
+ + " UPDATE SET salary = s.salary "
+ + "WHEN NOT MATCHED THEN "
+ + " INSERT *",
+ commitTarget());
+
+ Table table = validationCatalog.loadTable(tableIdent);
+
+ if (mode(table) == COPY_ON_WRITE) {
+ checkJoinAndFilterConditions(
+ query,
+ "ShuffledHashJoinExecTransformer [id], [id], FullOuter",
+ "((dep = 'd1' AND sub_dep IN ('sd1', 'sd3')) OR (dep = 'd6' AND sub_dep IN ('sd2', 'sd6')))");
+ } else {
+ checkJoinAndFilterConditions(
+ query,
+ "ShuffledHashJoinExecTransformer [id], [id], RightOuter",
+ "((dep = 'd1' AND sub_dep IN ('sd1', 'sd3')) OR (dep = 'd6' AND sub_dep IN ('sd2', 'sd6')))");
+ }
+
+ assertEquals(
+ "Should have expected rows",
+ ImmutableList.of(
+ row(1, 101, "d1", "sd1"), // updated
+ row(2, 200, "d2", "sd2"), // new
+ row(3, 300, "d3", "sd3"), // new
+ row(6, 600, "d6", "sd6")), // existing
+ sql("SELECT * FROM %s ORDER BY id", selectTarget()));
+ }
+
+ private void checkJoinAndFilterConditions(String query, String join, String icebergFilters) {
+ // disable runtime filtering for easier validation
+ withSQLConf(
+ ImmutableMap.of(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED().key(), "false"),
+ () -> {
+ SparkPlan sparkPlan = executeAndKeepPlan(() -> sql(query));
+ String planAsString = sparkPlan.toString().replaceAll("#(\\d+L?)", "");
+
+ // Remove "\n" because gluten prints BuildRight or BuildLeft in the end.
+ assertThat(planAsString).as("Join should match").contains(join);
+
+ assertThat(planAsString)
+ .as("Pushed filters must match")
+ .contains("[filters=" + icebergFilters + ",");
+ });
+ }
+
+ private RowLevelOperationMode mode(Table table) {
+ String modeName = table.properties().getOrDefault(MERGE_MODE, MERGE_MODE_DEFAULT);
+ return RowLevelOperationMode.fromName(modeName);
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadUpdate.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadUpdate.java
new file mode 100644
index 0000000000..f2db135cec
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenMergeOnReadUpdate.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extensions;
+
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.spark.extensions.TestMergeOnReadUpdate;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+public class TestGlutenMergeOnReadUpdate extends TestMergeOnReadUpdate {
+ public TestGlutenMergeOnReadUpdate(
+ String catalogName,
+ String implementation,
+ Map<String, String> config,
+ String fileFormat,
+ boolean vectorized,
+ String distributionMode,
+ boolean fanoutEnabled,
+ String branch,
+ PlanningMode planningMode) {
+ super(
+ catalogName,
+ implementation,
+ config,
+ fileFormat,
+ vectorized,
+ distributionMode,
+ fanoutEnabled,
+ branch,
+ planningMode);
+ }
+
+ @Test
+ public synchronized void testUpdateWithConcurrentTableRefresh() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testUpdateWithSerializableIsolation() {
+ System.out.println("Run timeout");
+ }
+
+ @Test
+ public synchronized void testUpdateWithSnapshotIsolation() throws ExecutionException {
+ System.out.println("Run timeout");
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenStoragePartitionedJoinsInRowLevelOperations.java
similarity index 62%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenStoragePartitionedJoinsInRowLevelOperations.java
index 632c806075..9d650c6f6c 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenStoragePartitionedJoinsInRowLevelOperations.java
@@ -14,13 +14,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.extensions;
-#pragma once
+import org.apache.iceberg.spark.extensions.TestStoragePartitionedJoinsInRowLevelOperations;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+import java.util.Map;
+
+public class TestGlutenStoragePartitionedJoinsInRowLevelOperations
+ extends TestStoragePartitionedJoinsInRowLevelOperations {
+ public TestGlutenStoragePartitionedJoinsInRowLevelOperations(
+ String catalogName, String implementation, Map<String, String> config) {
+ super(catalogName, implementation, config);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenSystemFunctionPushDownDQL.java
similarity index 66%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenSystemFunctionPushDownDQL.java
index 632c806075..059da14725 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenSystemFunctionPushDownDQL.java
@@ -14,13 +14,15 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.extensions;
-#pragma once
+import org.apache.iceberg.spark.extensions.TestSystemFunctionPushDownDQL;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+import java.util.Map;
+
+public class TestGlutenSystemFunctionPushDownDQL extends TestSystemFunctionPushDownDQL {
+ public TestGlutenSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config) {
+ super(catalogName, implementation, config);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenSystemFunctionPushDownInRowLevelOperations.java
similarity index 68%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenSystemFunctionPushDownInRowLevelOperations.java
index 632c806075..2eaaa6e5fe 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/extensions/TestGlutenSystemFunctionPushDownInRowLevelOperations.java
@@ -14,13 +14,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.extensions;
-#pragma once
+import java.util.Map;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenSystemFunctionPushDownInRowLevelOperations
+ extends TestGlutenSystemFunctionPushDownDQL {
+ public TestGlutenSystemFunctionPushDownInRowLevelOperations(
+ String catalogName, String implementation, Map<String, String> config) {
+ super(catalogName, implementation, config);
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestDataFrameWrites.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestDataFrameWrites.java
new file mode 100644
index 0000000000..678cec58d9
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestDataFrameWrites.java
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.source;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.*;
+import org.apache.iceberg.avro.Avro;
+import org.apache.iceberg.avro.AvroIterable;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.shaded.org.apache.avro.generic.GenericData.Record;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.spark.SparkSchemaUtil;
+import org.apache.iceberg.spark.SparkWriteOptions;
+import org.apache.iceberg.spark.data.AvroDataTest;
+import org.apache.iceberg.spark.data.RandomData;
+import org.apache.iceberg.spark.data.SparkAvroReader;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.SparkException;
+import org.apache.spark.TaskContext;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.function.MapPartitionsFunction;
+import org.apache.spark.sql.*;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.encoders.RowEncoder;
+import org.junit.*;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.*;
+
+import static org.apache.iceberg.spark.SparkSchemaUtil.convert;
+import static org.apache.iceberg.spark.data.TestHelpers.assertEqualsSafe;
+import static org.apache.iceberg.spark.data.TestHelpers.assertEqualsUnsafe;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+@RunWith(Parameterized.class)
+public class TestDataFrameWrites extends AvroDataTest {
+ private static final Configuration CONF = new Configuration();
+
+ private final String format;
+
+ @Parameterized.Parameters(name = "format = {0}")
+ public static Object[] parameters() {
+ return new Object[] {"parquet", "avro", "orc"};
+ }
+
+ public TestDataFrameWrites(String format) {
+ this.format = format;
+ }
+
+ private static SparkSession spark = null;
+ private static JavaSparkContext sc = null;
+
+ private Map<String, String> tableProperties;
+
+ private final org.apache.spark.sql.types.StructType sparkSchema =
+ new org.apache.spark.sql.types.StructType(
+ new org.apache.spark.sql.types.StructField[] {
+ new org.apache.spark.sql.types.StructField(
+ "optionalField",
+ org.apache.spark.sql.types.DataTypes.StringType,
+ true,
+ org.apache.spark.sql.types.Metadata.empty()),
+ new org.apache.spark.sql.types.StructField(
+ "requiredField",
+ org.apache.spark.sql.types.DataTypes.StringType,
+ false,
+ org.apache.spark.sql.types.Metadata.empty())
+ });
+
+ private final Schema icebergSchema =
+ new Schema(
+ Types.NestedField.optional(1, "optionalField", Types.StringType.get()),
+ Types.NestedField.required(2, "requiredField", Types.StringType.get()));
+
+ private final List<String> data0 =
+ Arrays.asList(
+ "{\"optionalField\": \"a1\", \"requiredField\": \"bid_001\"}",
+ "{\"optionalField\": \"a2\", \"requiredField\": \"bid_002\"}");
+ private final List<String> data1 =
+ Arrays.asList(
+ "{\"optionalField\": \"d1\", \"requiredField\": \"bid_101\"}",
+ "{\"optionalField\": \"d2\", \"requiredField\": \"bid_102\"}",
+ "{\"optionalField\": \"d3\", \"requiredField\": \"bid_103\"}",
+ "{\"optionalField\": \"d4\", \"requiredField\": \"bid_104\"}");
+
+ @BeforeClass
+ public static void startSpark() {
+ TestDataFrameWrites.spark = SparkSession.builder().master("local[2]").getOrCreate();
+ TestDataFrameWrites.sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
+ }
+
+ @AfterClass
+ public static void stopSpark() {
+ SparkSession currentSpark = TestDataFrameWrites.spark;
+ TestDataFrameWrites.spark = null;
+ TestDataFrameWrites.sc = null;
+ currentSpark.stop();
+ }
+
+ @Override
+ protected void writeAndValidate(Schema schema) throws IOException {
+ File location = createTableFolder();
+ Table table = createTable(schema, location);
+ writeAndValidateWithLocations(table, location, new File(location, "data"));
+ }
+
+ @Test
+ public void testWriteWithCustomDataLocation() throws IOException {
+ File location = createTableFolder();
+ File tablePropertyDataLocation = temp.newFolder("test-table-property-data-dir");
+ Table table = createTable(new Schema(SUPPORTED_PRIMITIVES.fields()), location);
+ table
+ .updateProperties()
+ .set(TableProperties.WRITE_DATA_LOCATION, tablePropertyDataLocation.getAbsolutePath())
+ .commit();
+ writeAndValidateWithLocations(table, location, tablePropertyDataLocation);
+ }
+
+ private File createTableFolder() throws IOException {
+ File parent = temp.newFolder("parquet");
+ File location = new File(parent, "test");
+ Assert.assertTrue("Mkdir should succeed", location.mkdirs());
+ return location;
+ }
+
+ private Table createTable(Schema schema, File location) {
+ HadoopTables tables = new HadoopTables(CONF);
+ return tables.create(schema, PartitionSpec.unpartitioned(), location.toString());
+ }
+
+ private void writeAndValidateWithLocations(Table table, File location, File expectedDataDir)
+ throws IOException {
+ Schema tableSchema = table.schema(); // use the table schema because ids are reassigned
+
+ table.updateProperties().set(TableProperties.DEFAULT_FILE_FORMAT, format).commit();
+
+ Iterable<Record> expected = RandomData.generate(tableSchema, 100, 0L);
+ writeData(expected, tableSchema, location.toString());
+
+ table.refresh();
+
+ List<Row> actual = readTable(location.toString());
+
+ Iterator<Record> expectedIter = expected.iterator();
+ Iterator<Row> actualIter = actual.iterator();
+ while (expectedIter.hasNext() && actualIter.hasNext()) {
+ assertEqualsSafe(tableSchema.asStruct(), expectedIter.next(), actualIter.next());
+ }
+ Assert.assertEquals(
+ "Both iterators should be exhausted", expectedIter.hasNext(), actualIter.hasNext());
+
+ table
+ .currentSnapshot()
+ .addedDataFiles(table.io())
+ .forEach(
+ dataFile ->
+ Assert.assertTrue(
+ String.format(
+ "File should have the parent directory %s, but has: %s.",
+ expectedDataDir.getAbsolutePath(), dataFile.path()),
+ URI.create(dataFile.path().toString())
+ .getPath()
+ .startsWith(expectedDataDir.getAbsolutePath())));
+ }
+
+ private List<Row> readTable(String location) {
+ Dataset<Row> result = spark.read().format("iceberg").load(location);
+
+ return result.collectAsList();
+ }
+
+ private void writeData(Iterable<Record> records, Schema schema, String location)
+ throws IOException {
+ Dataset<Row> df = createDataset(records, schema);
+ DataFrameWriter<?> writer = df.write().format("iceberg").mode("append");
+ writer.save(location);
+ }
+
+ private void writeDataWithFailOnPartition(
+ Iterable<Record> records, Schema schema, String location) throws IOException, SparkException {
+ final int numPartitions = 10;
+ final int partitionToFail = new Random().nextInt(numPartitions);
+ MapPartitionsFunction<Row, Row> failOnFirstPartitionFunc =
+ input -> {
+ int partitionId = TaskContext.getPartitionId();
+
+ if (partitionId == partitionToFail) {
+ throw new SparkException(
+ String.format("Intended exception in partition %d !", partitionId));
+ }
+ return input;
+ };
+
+ Dataset<Row> df =
+ createDataset(records, schema)
+ .repartition(numPartitions)
+ .mapPartitions(failOnFirstPartitionFunc, RowEncoder.apply(convert(schema)));
+ // This trick is needed because Spark 3 handles decimal overflow in RowEncoder which "changes"
+ // nullability of the column to "true" regardless of original nullability.
+ // Setting "check-nullability" option to "false" doesn't help as it fails at Spark analyzer.
+ Dataset<Row> convertedDf = df.sqlContext().createDataFrame(df.rdd(), convert(schema));
+ DataFrameWriter<?> writer = convertedDf.write().format("iceberg").mode("append");
+ writer.save(location);
+ }
+
+ private Dataset<Row> createDataset(Iterable<Record> records, Schema schema) throws IOException {
+ // this uses the SparkAvroReader to create a DataFrame from the list of records
+ // it assumes that SparkAvroReader is correct
+ File testFile = temp.newFile();
+ Assert.assertTrue("Delete should succeed", testFile.delete());
+
+ try (FileAppender<Record> writer =
+ Avro.write(Files.localOutput(testFile)).schema(schema).named("test").build()) {
+ for (Record rec : records) {
+ writer.add(rec);
+ }
+ }
+
+ // make sure the dataframe matches the records before moving on
+ List<InternalRow> rows = Lists.newArrayList();
+ try (AvroIterable<InternalRow> reader =
+ Avro.read(Files.localInput(testFile))
+ .createReaderFunc(SparkAvroReader::new)
+ .project(schema)
+ .build()) {
+
+ Iterator<Record> recordIter = records.iterator();
+ Iterator<InternalRow> readIter = reader.iterator();
+ while (recordIter.hasNext() && readIter.hasNext()) {
+ InternalRow row = readIter.next();
+ assertEqualsUnsafe(schema.asStruct(), recordIter.next(), row);
+ rows.add(row);
+ }
+ Assert.assertEquals(
+ "Both iterators should be exhausted", recordIter.hasNext(), readIter.hasNext());
+ }
+
+ JavaRDD<InternalRow> rdd = sc.parallelize(rows);
+ return spark.internalCreateDataFrame(JavaRDD.toRDD(rdd), convert(schema), false);
+ }
+
+ @Test
+ public void testNullableWithWriteOption() throws IOException {
+ Assume.assumeTrue(
+ "Spark 3 rejects writing nulls to a required column", spark.version().startsWith("2"));
+
+ File location = new File(temp.newFolder("parquet"), "test");
+ String sourcePath = String.format("%s/nullable_poc/sourceFolder/", location);
+ String targetPath = String.format("%s/nullable_poc/targetFolder/", location);
+
+ tableProperties = ImmutableMap.of(TableProperties.WRITE_DATA_LOCATION, targetPath);
+
+ // read this and append to iceberg dataset
+ spark
+ .read()
+ .schema(sparkSchema)
+ .json(JavaSparkContext.fromSparkContext(spark.sparkContext()).parallelize(data1))
+ .write()
+ .parquet(sourcePath);
+
+ // this is our iceberg dataset to which we will append data
+ new HadoopTables(spark.sessionState().newHadoopConf())
+ .create(
+ icebergSchema,
+ PartitionSpec.builderFor(icebergSchema).identity("requiredField").build(),
+ tableProperties,
+ targetPath);
+
+ // this is the initial data inside the iceberg dataset
+ spark
+ .read()
+ .schema(sparkSchema)
+ .json(JavaSparkContext.fromSparkContext(spark.sparkContext()).parallelize(data0))
+ .write()
+ .format("iceberg")
+ .mode(SaveMode.Append)
+ .save(targetPath);
+
+ // read from parquet and append to iceberg w/ nullability check disabled
+ spark
+ .read()
+ .schema(SparkSchemaUtil.convert(icebergSchema))
+ .parquet(sourcePath)
+ .write()
+ .format("iceberg")
+ .option(SparkWriteOptions.CHECK_NULLABILITY, false)
+ .mode(SaveMode.Append)
+ .save(targetPath);
+
+ // read all data
+ List<Row> rows = spark.read().format("iceberg").load(targetPath).collectAsList();
+ Assert.assertEquals("Should contain 6 rows", 6, rows.size());
+ }
+
+ @Test
+ public void testNullableWithSparkSqlOption() throws IOException {
+ Assume.assumeTrue(
+ "Spark 3 rejects writing nulls to a required column", spark.version().startsWith("2"));
+
+ File location = new File(temp.newFolder("parquet"), "test");
+ String sourcePath = String.format("%s/nullable_poc/sourceFolder/", location);
+ String targetPath = String.format("%s/nullable_poc/targetFolder/", location);
+
+ tableProperties = ImmutableMap.of(TableProperties.WRITE_DATA_LOCATION, targetPath);
+
+ // read this and append to iceberg dataset
+ spark
+ .read()
+ .schema(sparkSchema)
+ .json(JavaSparkContext.fromSparkContext(spark.sparkContext()).parallelize(data1))
+ .write()
+ .parquet(sourcePath);
+
+ SparkSession newSparkSession =
+ SparkSession.builder()
+ .master("local[2]")
+ .appName("NullableTest")
+ .config(SparkSQLProperties.CHECK_NULLABILITY, false)
+ .getOrCreate();
+
+ // this is our iceberg dataset to which we will append data
+ new HadoopTables(newSparkSession.sessionState().newHadoopConf())
+ .create(
+ icebergSchema,
+ PartitionSpec.builderFor(icebergSchema).identity("requiredField").build(),
+ tableProperties,
+ targetPath);
+
+ // this is the initial data inside the iceberg dataset
+ newSparkSession
+ .read()
+ .schema(sparkSchema)
+ .json(JavaSparkContext.fromSparkContext(spark.sparkContext()).parallelize(data0))
+ .write()
+ .format("iceberg")
+ .mode(SaveMode.Append)
+ .save(targetPath);
+
+ // read from parquet and append to iceberg
+ newSparkSession
+ .read()
+ .schema(SparkSchemaUtil.convert(icebergSchema))
+ .parquet(sourcePath)
+ .write()
+ .format("iceberg")
+ .mode(SaveMode.Append)
+ .save(targetPath);
+
+ // read all data
+ List<Row> rows = newSparkSession.read().format("iceberg").load(targetPath).collectAsList();
+ Assert.assertEquals("Should contain 6 rows", 6, rows.size());
+ }
+
+ @Test
+ public void testFaultToleranceOnWrite() throws IOException {
+ File location = createTableFolder();
+ Schema schema = new Schema(SUPPORTED_PRIMITIVES.fields());
+ Table table = createTable(schema, location);
+
+ Iterable<Record> records = RandomData.generate(schema, 100, 0L);
+ writeData(records, schema, location.toString());
+
+ table.refresh();
+
+ Snapshot snapshotBeforeFailingWrite = table.currentSnapshot();
+ List<Row> resultBeforeFailingWrite = readTable(location.toString());
+
+ Iterable<Record> records2 = RandomData.generate(schema, 100, 0L);
+
+ assertThatThrownBy(() -> writeDataWithFailOnPartition(records2, schema, location.toString()))
+ .isInstanceOf(SparkException.class);
+
+ table.refresh();
+
+ Snapshot snapshotAfterFailingWrite = table.currentSnapshot();
+ List<Row> resultAfterFailingWrite = readTable(location.toString());
+
+ Assert.assertEquals(snapshotAfterFailingWrite, snapshotBeforeFailingWrite);
+ Assert.assertEquals(resultAfterFailingWrite, resultBeforeFailingWrite);
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestFilteredScan.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestFilteredScan.java
new file mode 100644
index 0000000000..5175862420
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestFilteredScan.java
@@ -0,0 +1,726 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.source;
+
+import org.apache.gluten.TestConfUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.GenericAppenderFactory;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.spark.SparkReadOptions;
+import org.apache.iceberg.spark.data.GenericsHelpers;
+import org.apache.iceberg.spark.source.GlutenSparkScanBuilder;
+import org.apache.iceberg.spark.source.SparkScanBuilder;
+import org.apache.iceberg.transforms.Transforms;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Encoders;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.api.java.UDF1;
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.ScanBuilder;
+import org.apache.spark.sql.connector.read.SupportsPushDownV2Filters;
+import org.apache.spark.sql.sources.And;
+import org.apache.spark.sql.sources.EqualTo;
+import org.apache.spark.sql.sources.Filter;
+import org.apache.spark.sql.sources.GreaterThan;
+import org.apache.spark.sql.sources.LessThan;
+import org.apache.spark.sql.sources.Not;
+import org.apache.spark.sql.sources.StringStartsWith;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.util.Arrays;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.iceberg.Files.localOutput;
+import static org.apache.iceberg.PlanningMode.DISTRIBUTED;
+import static org.apache.iceberg.PlanningMode.LOCAL;
+import static org.apache.spark.sql.catalyst.util.DateTimeUtils.fromJavaTimestamp;
+import static org.apache.spark.sql.functions.callUDF;
+import static org.apache.spark.sql.functions.column;
+import static org.assertj.core.api.Assertions.assertThat;
+
+@SuppressWarnings("checkstyle:LineLegth")
+// Timestamp with timezone for orc format is bot supported, but cannot fallback in
+// BatchScanTransformer
+// because we cannot distinguish file format but can only get the schema
+// Error Source: RUNTIME
+// Error Code: INVALID_STATE
+// Reason: TIMESTAMP_INSTANT not supported yet.
+// Retriable: False
+// Context: Split [Hive:
+// /var/folders/63/845y6pk53dx_83hpw8ztdchw0000gn/T/junit10976573641877215189/TestFilteredScan/unpartitioned/data/b464438c-e706-412b-bcc9-71321ff4aead.orc 0 - 629] Task Gluten_Stage_6_TID_6_VTID_4
+// Additional Context: Operator: TableScan[0] 0
+// Function: kind
+// File: code/gluten/ep/build-bolt/build/bolt_ep/bolt/dwio/dwrf/common/FileMetadata.cpp
+// Line: 107
+// Stack trace:
+@RunWith(Parameterized.class)
+public class TestFilteredScan {
+ private static final Configuration CONF = new Configuration();
+ private static final HadoopTables TABLES = new HadoopTables(CONF);
+
+ private static final Schema SCHEMA =
+ new Schema(
+ Types.NestedField.required(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "ts", Types.TimestampType.withZone()),
+ Types.NestedField.optional(3, "data", Types.StringType.get()));
+
+ private static final PartitionSpec BUCKET_BY_ID =
+ PartitionSpec.builderFor(SCHEMA).bucket("id", 4).build();
+
+ private static final PartitionSpec PARTITION_BY_DAY =
+ PartitionSpec.builderFor(SCHEMA).day("ts").build();
+
+ private static final PartitionSpec PARTITION_BY_HOUR =
+ PartitionSpec.builderFor(SCHEMA).hour("ts").build();
+
+ private static final PartitionSpec PARTITION_BY_DATA =
+ PartitionSpec.builderFor(SCHEMA).identity("data").build();
+
+ private static final PartitionSpec PARTITION_BY_ID =
+ PartitionSpec.builderFor(SCHEMA).identity("id").build();
+
+ private static SparkSession spark = null;
+
+ @BeforeClass
+ public static void startSpark() {
+ TestFilteredScan.spark =
+ SparkSession.builder().master("local[2]").config(TestConfUtil.GLUTEN_CONF).getOrCreate();
+
+ // define UDFs used by partition tests
+ Function<Object, Integer> bucket4 = Transforms.bucket(4).bind(Types.LongType.get());
+ spark.udf().register("bucket4", (UDF1<Long, Integer>) bucket4::apply, IntegerType$.MODULE$);
+
+ Function<Object, Integer> day = Transforms.day().bind(Types.TimestampType.withZone());
+ spark
+ .udf()
+ .register(
+ "ts_day",
+ (UDF1<Timestamp, Integer>) timestamp -> day.apply(fromJavaTimestamp(timestamp)),
+ IntegerType$.MODULE$);
+
+ Function<Object, Integer> hour = Transforms.hour().bind(Types.TimestampType.withZone());
+ spark
+ .udf()
+ .register(
+ "ts_hour",
+ (UDF1<Timestamp, Integer>) timestamp -> hour.apply(fromJavaTimestamp(timestamp)),
+ IntegerType$.MODULE$);
+
+ spark.udf().register("data_ident", (UDF1<String, String>) data -> data, StringType$.MODULE$);
+ spark.udf().register("id_ident", (UDF1<Long, Long>) id -> id, LongType$.MODULE$);
+ }
+
+ @AfterClass
+ public static void stopSpark() {
+ SparkSession currentSpark = TestFilteredScan.spark;
+ TestFilteredScan.spark = null;
+ currentSpark.stop();
+ }
+
+ @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+ private final String format;
+ private final boolean vectorized;
+ private final PlanningMode planningMode;
+
+ @Parameterized.Parameters(name = "format = {0}, vectorized = {1}, planningMode = {2}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {"parquet", false, LOCAL},
+ {"parquet", true, DISTRIBUTED},
+ {"avro", false, LOCAL},
+ // TODO, support orc timestamp with time zone
+ // {"orc", false, DISTRIBUTED},
+ // {"orc", true, LOCAL}
+ };
+ }
+
+ public TestFilteredScan(String format, boolean vectorized, PlanningMode planningMode) {
+ this.format = format;
+ this.vectorized = vectorized;
+ this.planningMode = planningMode;
+ }
+
+ private File parent = null;
+ private File unpartitioned = null;
+ private List<Record> records = null;
+
+ @Before
+ public void writeUnpartitionedTable() throws IOException {
+ this.parent = temp.newFolder("TestFilteredScan");
+ this.unpartitioned = new File(parent, "unpartitioned");
+ File dataFolder = new File(unpartitioned, "data");
+ Assert.assertTrue("Mkdir should succeed", dataFolder.mkdirs());
+
+ Table table =
+ TABLES.create(
+ SCHEMA,
+ PartitionSpec.unpartitioned(),
+ ImmutableMap.of(
+ TableProperties.DATA_PLANNING_MODE,
+ planningMode.modeName(),
+ TableProperties.DELETE_PLANNING_MODE,
+ planningMode.modeName()),
+ unpartitioned.toString());
+ Schema tableSchema = table.schema(); // use the table schema because ids are reassigned
+
+ FileFormat fileFormat = FileFormat.fromString(format);
+
+ File testFile = new File(dataFolder, fileFormat.addExtension(UUID.randomUUID().toString()));
+
+ this.records = testRecords(tableSchema);
+
+ try (FileAppender<Record> writer =
+ new GenericAppenderFactory(tableSchema).newAppender(localOutput(testFile), fileFormat)) {
+ writer.addAll(records);
+ }
+
+ DataFile file =
+ DataFiles.builder(PartitionSpec.unpartitioned())
+ .withRecordCount(records.size())
+ .withFileSizeInBytes(testFile.length())
+ .withPath(testFile.toString())
+ .build();
+
+ table.newAppend().appendFile(file).commit();
+ }
+
+ @Test
+ public void testUnpartitionedIDFilters() {
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", unpartitioned.toString()));
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ for (int i = 0; i < 10; i += 1) {
+ pushFilters(builder, EqualTo.apply("id", i));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] partitions = scan.planInputPartitions();
+ Assert.assertEquals("Should only create one task for a small file", 1, partitions.length);
+
+ // validate row filtering
+ assertEqualsSafe(
+ SCHEMA.asStruct(), expected(i), read(unpartitioned.toString(), vectorized, "id = " + i));
+ }
+ }
+
+ @Test
+ public void testUnpartitionedCaseInsensitiveIDFilters() {
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", unpartitioned.toString()));
+
+ // set spark.sql.caseSensitive to false
+ String caseSensitivityBeforeTest = TestFilteredScan.spark.conf().get("spark.sql.caseSensitive");
+ TestFilteredScan.spark.conf().set("spark.sql.caseSensitive", "false");
+
+ try {
+
+ for (int i = 0; i < 10; i += 1) {
+ SparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options)
+ .caseSensitive(false);
+
+ pushFilters(
+ builder,
+ EqualTo.apply("ID", i)); // note lower(ID) == lower(id), so there must be a match
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+ Assert.assertEquals("Should only create one task for a small file", 1, tasks.length);
+
+ // validate row filtering
+ assertEqualsSafe(
+ SCHEMA.asStruct(),
+ expected(i),
+ read(unpartitioned.toString(), vectorized, "id = " + i));
+ }
+ } finally {
+ // return global conf to previous state
+ TestFilteredScan.spark.conf().set("spark.sql.caseSensitive", caseSensitivityBeforeTest);
+ }
+ }
+
+ @Test
+ public void testUnpartitionedTimestampFilter() {
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", unpartitioned.toString()));
+
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, LessThan.apply("ts", "2017-12-22T00:00:00+00:00"));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+ Assert.assertEquals("Should only create one task for a small file", 1, tasks.length);
+
+ assertEqualsSafe(
+ SCHEMA.asStruct(),
+ expected(5, 6, 7, 8, 9),
+ read(
+ unpartitioned.toString(),
+ vectorized,
+ "ts < cast('2017-12-22 00:00:00+00:00' as timestamp)"));
+ }
+
+ @Test
+ public void testBucketPartitionedIDFilters() {
+ Table table = buildPartitionedTable("bucketed_by_id", BUCKET_BY_ID, "bucket4", "id");
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+
+ Batch unfiltered =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options)
+ .build()
+ .toBatch();
+ Assert.assertEquals(
+ "Unfiltered table should created 4 read tasks", 4, unfiltered.planInputPartitions().length);
+
+ for (int i = 0; i < 10; i += 1) {
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, EqualTo.apply("id", i));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+
+ // validate predicate push-down
+ Assert.assertEquals("Should create one task for a single bucket", 1, tasks.length);
+
+ // validate row filtering
+ assertEqualsSafe(
+ SCHEMA.asStruct(), expected(i), read(table.location(), vectorized, "id = " + i));
+ }
+ }
+
+ @SuppressWarnings("checkstyle:AvoidNestedBlocks")
+ @Test
+ public void testDayPartitionedTimestampFilters() {
+ Table table = buildPartitionedTable("partitioned_by_day", PARTITION_BY_DAY, "ts_day", "ts");
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+ Batch unfiltered =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options)
+ .build()
+ .toBatch();
+
+ Assert.assertEquals(
+ "Unfiltered table should created 2 read tasks", 2, unfiltered.planInputPartitions().length);
+
+ {
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, LessThan.apply("ts", "2017-12-22T00:00:00+00:00"));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+ Assert.assertEquals("Should create one task for 2017-12-21", 1, tasks.length);
+
+ assertEqualsSafe(
+ SCHEMA.asStruct(),
+ expected(5, 6, 7, 8, 9),
+ read(
+ table.location(), vectorized, "ts < cast('2017-12-22 00:00:00+00:00' as timestamp)"));
+ }
+
+ {
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(
+ builder,
+ And.apply(
+ GreaterThan.apply("ts", "2017-12-22T06:00:00+00:00"),
+ LessThan.apply("ts", "2017-12-22T08:00:00+00:00")));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+ Assert.assertEquals("Should create one task for 2017-12-22", 1, tasks.length);
+
+ assertEqualsSafe(
+ SCHEMA.asStruct(),
+ expected(1, 2),
+ read(
+ table.location(),
+ vectorized,
+ "ts > cast('2017-12-22 06:00:00+00:00' as timestamp) and "
+ + "ts < cast('2017-12-22 08:00:00+00:00' as timestamp)"));
+ }
+ }
+
+ @SuppressWarnings("checkstyle:AvoidNestedBlocks")
+ @Test
+ public void testHourPartitionedTimestampFilters() {
+ Table table = buildPartitionedTable("partitioned_by_hour", PARTITION_BY_HOUR, "ts_hour", "ts");
+
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+ Batch unfiltered =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options)
+ .build()
+ .toBatch();
+
+ Assert.assertEquals(
+ "Unfiltered table should created 9 read tasks", 9, unfiltered.planInputPartitions().length);
+
+ {
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, LessThan.apply("ts", "2017-12-22T00:00:00+00:00"));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+ Assert.assertEquals("Should create 4 tasks for 2017-12-21: 15, 17, 21, 22", 4, tasks.length);
+
+ assertEqualsSafe(
+ SCHEMA.asStruct(),
+ expected(8, 9, 7, 6, 5),
+ read(
+ table.location(), vectorized, "ts < cast('2017-12-22 00:00:00+00:00' as timestamp)"));
+ }
+
+ {
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(
+ builder,
+ And.apply(
+ GreaterThan.apply("ts", "2017-12-22T06:00:00+00:00"),
+ LessThan.apply("ts", "2017-12-22T08:00:00+00:00")));
+ Batch scan = builder.build().toBatch();
+
+ InputPartition[] tasks = scan.planInputPartitions();
+ Assert.assertEquals("Should create 2 tasks for 2017-12-22: 6, 7", 2, tasks.length);
+
+ assertEqualsSafe(
+ SCHEMA.asStruct(),
+ expected(2, 1),
+ read(
+ table.location(),
+ vectorized,
+ "ts > cast('2017-12-22 06:00:00+00:00' as timestamp) and "
+ + "ts < cast('2017-12-22 08:00:00+00:00' as timestamp)"));
+ }
+ }
+
+ @SuppressWarnings("checkstyle:AvoidNestedBlocks")
+ @Test
+ public void testFilterByNonProjectedColumn() {
+ {
+ Schema actualProjection = SCHEMA.select("id", "data");
+ List<Record> expected = Lists.newArrayList();
+ for (Record rec : expected(5, 6, 7, 8, 9)) {
+ expected.add(projectFlat(actualProjection, rec));
+ }
+
+ assertEqualsSafe(
+ actualProjection.asStruct(),
+ expected,
+ read(
+ unpartitioned.toString(),
+ vectorized,
+ "ts < cast('2017-12-22 00:00:00+00:00' as timestamp)",
+ "id",
+ "data"));
+ }
+
+ {
+ // only project id: ts will be projected because of the filter, but data will not be included
+
+ Schema actualProjection = SCHEMA.select("id");
+ List<Record> expected = Lists.newArrayList();
+ for (Record rec : expected(1, 2)) {
+ expected.add(projectFlat(actualProjection, rec));
+ }
+
+ assertEqualsSafe(
+ actualProjection.asStruct(),
+ expected,
+ read(
+ unpartitioned.toString(),
+ vectorized,
+ "ts > cast('2017-12-22 06:00:00+00:00' as timestamp) and "
+ + "ts < cast('2017-12-22 08:00:00+00:00' as timestamp)",
+ "id"));
+ }
+ }
+
+ @Test
+ public void testPartitionedByDataStartsWithFilter() {
+ Table table =
+ buildPartitionedTable("partitioned_by_data", PARTITION_BY_DATA, "data_ident", "data");
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, new StringStartsWith("data", "junc"));
+ Batch scan = builder.build().toBatch();
+
+ Assert.assertEquals(1, scan.planInputPartitions().length);
+ }
+
+ @Test
+ public void testPartitionedByDataNotStartsWithFilter() {
+ Table table =
+ buildPartitionedTable("partitioned_by_data", PARTITION_BY_DATA, "data_ident", "data");
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, new Not(new StringStartsWith("data", "junc")));
+ Batch scan = builder.build().toBatch();
+
+ Assert.assertEquals(9, scan.planInputPartitions().length);
+ }
+
+ @Test
+ public void testPartitionedByIdStartsWith() {
+ Table table = buildPartitionedTable("partitioned_by_id", PARTITION_BY_ID, "id_ident", "id");
+
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, new StringStartsWith("data", "junc"));
+ Batch scan = builder.build().toBatch();
+
+ Assert.assertEquals(1, scan.planInputPartitions().length);
+ }
+
+ @Test
+ public void testPartitionedByIdNotStartsWith() {
+ Table table = buildPartitionedTable("partitioned_by_id", PARTITION_BY_ID, "id_ident", "id");
+
+ CaseInsensitiveStringMap options =
+ new CaseInsensitiveStringMap(ImmutableMap.of("path", table.location()));
+
+ GlutenSparkScanBuilder builder =
+ new GlutenSparkScanBuilder(spark, TABLES.load(options.get("path")), options);
+
+ pushFilters(builder, new Not(new StringStartsWith("data", "junc")));
+ Batch scan = builder.build().toBatch();
+
+ Assert.assertEquals(9, scan.planInputPartitions().length);
+ }
+
+ @Test
+ public void testUnpartitionedStartsWith() {
+ Dataset<Row> df =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(unpartitioned.toString());
+
+ List<String> matchedData =
+ df.select("data").where("data LIKE 'jun%'").as(Encoders.STRING()).collectAsList();
+
+ Assert.assertEquals(1, matchedData.size());
+ Assert.assertEquals("junction", matchedData.get(0));
+ }
+
+ @Test
+ public void testUnpartitionedNotStartsWith() {
+ Dataset<Row> df =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(unpartitioned.toString());
+
+ List<String> matchedData =
+ df.select("data").where("data NOT LIKE 'jun%'").as(Encoders.STRING()).collectAsList();
+
+ List<String> expected =
+ testRecords(SCHEMA).stream()
+ .map(r -> r.getField("data").toString())
+ .filter(d -> !d.startsWith("jun"))
+ .collect(Collectors.toList());
+
+ Assert.assertEquals(9, matchedData.size());
+ Assert.assertEquals(Sets.newHashSet(expected), Sets.newHashSet(matchedData));
+ }
+
+ private static Record projectFlat(Schema projection, Record record) {
+ Record result = GenericRecord.create(projection);
+ List<Types.NestedField> fields = projection.asStruct().fields();
+ for (int i = 0; i < fields.size(); i += 1) {
+ Types.NestedField field = fields.get(i);
+ result.set(i, record.getField(field.name()));
+ }
+ return result;
+ }
+
+ public static void assertEqualsUnsafe(
+ Types.StructType struct, List<Record> expected, List<UnsafeRow> actual) {
+ // TODO: match records by ID
+ int numRecords = Math.min(expected.size(), actual.size());
+ for (int i = 0; i < numRecords; i += 1) {
+ GenericsHelpers.assertEqualsUnsafe(struct, expected.get(i), actual.get(i));
+ }
+ Assert.assertEquals("Number of results should match expected", expected.size(), actual.size());
+ }
+
+ public static void assertEqualsSafe(
+ Types.StructType struct, List<Record> expected, List<Row> actual) {
+ // TODO: match records by ID
+ int numRecords = Math.min(expected.size(), actual.size());
+ for (int i = 0; i < numRecords; i += 1) {
+ GenericsHelpers.assertEqualsSafe(struct, expected.get(i), actual.get(i));
+ }
+ Assert.assertEquals("Number of results should match expected", expected.size(), actual.size());
+ }
+
+ private List<Record> expected(int... ordinals) {
+ List<Record> expected = Lists.newArrayListWithExpectedSize(ordinals.length);
+ for (int ord : ordinals) {
+ expected.add(records.get(ord));
+ }
+ return expected;
+ }
+
+ private void pushFilters(ScanBuilder scan, Filter... filters) {
+ assertThat(scan).isInstanceOf(SupportsPushDownV2Filters.class);
+ SupportsPushDownV2Filters filterable = (SupportsPushDownV2Filters) scan;
+ filterable.pushPredicates(Arrays.stream(filters).map(Filter::toV2).toArray(Predicate[]::new));
+ }
+
+ private Table buildPartitionedTable(
+ String desc, PartitionSpec spec, String udf, String partitionColumn) {
+ File location = new File(parent, desc);
+ Table table = TABLES.create(SCHEMA, spec, location.toString());
+
+ // Do not combine or split files because the tests expect a split per partition.
+ // A target split size of 2048 helps us achieve that.
+ table.updateProperties().set("read.split.target-size", "2048").commit();
+
+ // copy the unpartitioned table into the partitioned table to produce the partitioned data
+ Dataset<Row> allRows =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(unpartitioned.toString());
+
+ allRows
+ .coalesce(1) // ensure only 1 file per partition is written
+ .withColumn("part", callUDF(udf, column(partitionColumn)))
+ .sortWithinPartitions("part")
+ .drop("part")
+ .write()
+ .format("iceberg")
+ .mode("append")
+ .save(table.location());
+
+ table.refresh();
+
+ return table;
+ }
+
+ private List<Record> testRecords(Schema schema) {
+ return Lists.newArrayList(
+ record(schema, 0L, parse("2017-12-22T09:20:44.294658+00:00"), "junction"),
+ record(schema, 1L, parse("2017-12-22T07:15:34.582910+00:00"), "alligator"),
+ record(schema, 2L, parse("2017-12-22T06:02:09.243857+00:00"), ""),
+ record(schema, 3L, parse("2017-12-22T03:10:11.134509+00:00"), "clapping"),
+ record(schema, 4L, parse("2017-12-22T00:34:00.184671+00:00"), "brush"),
+ record(schema, 5L, parse("2017-12-21T22:20:08.935889+00:00"), "trap"),
+ record(schema, 6L, parse("2017-12-21T21:55:30.589712+00:00"), "element"),
+ record(schema, 7L, parse("2017-12-21T17:31:14.532797+00:00"), "limited"),
+ record(schema, 8L, parse("2017-12-21T15:21:51.237521+00:00"), "global"),
+ record(schema, 9L, parse("2017-12-21T15:02:15.230570+00:00"), "goldfish"));
+ }
+
+ private static List<Row> read(String table, boolean vectorized, String expr) {
+ return read(table, vectorized, expr, "*");
+ }
+
+ private static List<Row> read(
+ String table, boolean vectorized, String expr, String select0, String... selectN) {
+ Dataset<Row> dataset =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(table)
+ .filter(expr)
+ .select(select0, selectN);
+ return dataset.collectAsList();
+ }
+
+ private static OffsetDateTime parse(String timestamp) {
+ return OffsetDateTime.parse(timestamp);
+ }
+
+ private static Record record(Schema schema, Object... values) {
+ Record rec = GenericRecord.create(schema);
+ for (int i = 0; i < values.length; i += 1) {
+ rec.set(i, values[i]);
+ }
+ return rec;
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestForwardCompatibility.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestForwardCompatibility.java
new file mode 100644
index 0000000000..3913ff4d61
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestForwardCompatibility.java
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.source;
+
+import org.apache.gluten.TestConfUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.*;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.shaded.org.apache.avro.generic.GenericData;
+import org.apache.iceberg.spark.data.RandomData;
+import org.apache.iceberg.spark.data.TestHelpers;
+import org.apache.iceberg.spark.source.SimpleRecord;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.*;
+import org.apache.spark.sql.execution.streaming.MemoryStream;
+import org.apache.spark.sql.streaming.StreamingQuery;
+import org.apache.spark.sql.streaming.StreamingQueryException;
+import org.junit.*;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeoutException;
+
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import static org.apache.iceberg.Files.localInput;
+import static org.apache.iceberg.Files.localOutput;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+public class TestForwardCompatibility {
+ private static final Configuration CONF = new Configuration();
+
+ private static final Schema SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ // create a spec for the schema that uses a "zero" transform that produces all 0s
+ private static final PartitionSpec UNKNOWN_SPEC =
+ org.apache.iceberg.TestHelpers.newExpectedSpecBuilder()
+ .withSchema(SCHEMA)
+ .withSpecId(0)
+ .addField("zero", 1, "id_zero")
+ .build();
+
+ // create a fake spec to use to write table metadata
+ private static final PartitionSpec FAKE_SPEC =
+ org.apache.iceberg.TestHelpers.newExpectedSpecBuilder()
+ .withSchema(SCHEMA)
+ .withSpecId(0)
+ .addField("identity", 1, "id_zero")
+ .build();
+
+ @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+ private static SparkSession spark = null;
+
+ @BeforeClass
+ public static void startSpark() {
+ TestForwardCompatibility.spark =
+ SparkSession.builder().master("local[2]").config(TestConfUtil.GLUTEN_CONF).getOrCreate();
+ }
+
+ @AfterClass
+ public static void stopSpark() {
+ SparkSession currentSpark = TestForwardCompatibility.spark;
+ TestForwardCompatibility.spark = null;
+ currentSpark.stop();
+ }
+
+ @Test
+ public void testSparkWriteFailsUnknownTransform() throws IOException {
+ File parent = temp.newFolder("avro");
+ File location = new File(parent, "test");
+ File dataFolder = new File(location, "data");
+ dataFolder.mkdirs();
+
+ HadoopTables tables = new HadoopTables(CONF);
+ tables.create(SCHEMA, UNKNOWN_SPEC, location.toString());
+
+ List<SimpleRecord> expected =
+ Lists.newArrayList(
+ new SimpleRecord(1, "a"), new SimpleRecord(2, "b"), new SimpleRecord(3, "c"));
+
+ Dataset<Row> df = spark.createDataFrame(expected, SimpleRecord.class);
+
+ assertThatThrownBy(
+ () ->
+ df.select("id", "data")
+ .write()
+ .format("iceberg")
+ .mode("append")
+ .save(location.toString()))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageEndingWith("Cannot write using unsupported transforms: zero");
+ }
+
+ @Test
+ public void testSparkStreamingWriteFailsUnknownTransform() throws IOException, TimeoutException {
+ File parent = temp.newFolder("avro");
+ File location = new File(parent, "test");
+ File dataFolder = new File(location, "data");
+ dataFolder.mkdirs();
+ File checkpoint = new File(parent, "checkpoint");
+ checkpoint.mkdirs();
+
+ HadoopTables tables = new HadoopTables(CONF);
+ tables.create(SCHEMA, UNKNOWN_SPEC, location.toString());
+
+ MemoryStream<Integer> inputStream = newMemoryStream(1, spark.sqlContext(), Encoders.INT());
+ StreamingQuery query =
+ inputStream
+ .toDF()
+ .selectExpr("value AS id", "CAST (value AS STRING) AS data")
+ .writeStream()
+ .outputMode("append")
+ .format("iceberg")
+ .option("checkpointLocation", checkpoint.toString())
+ .option("path", location.toString())
+ .start();
+
+ List<Integer> batch1 = Lists.newArrayList(1, 2);
+ send(batch1, inputStream);
+
+ assertThatThrownBy(query::processAllAvailable)
+ .isInstanceOf(StreamingQueryException.class)
+ .hasMessageEndingWith("Cannot write using unsupported transforms: zero");
+ }
+
+ @Test
+ public void testSparkCanReadUnknownTransform() throws IOException {
+ File parent = temp.newFolder("avro");
+ File location = new File(parent, "test");
+ File dataFolder = new File(location, "data");
+ dataFolder.mkdirs();
+
+ HadoopTables tables = new HadoopTables(CONF);
+ Table table = tables.create(SCHEMA, UNKNOWN_SPEC, location.toString());
+
+ // enable snapshot inheritance to avoid rewriting the manifest with an unknown transform
+ table.updateProperties().set(TableProperties.SNAPSHOT_ID_INHERITANCE_ENABLED, "true").commit();
+
+ List<GenericData.Record> expected = RandomData.generateList(table.schema(), 100, 1L);
+
+ File parquetFile =
+ new File(dataFolder, FileFormat.PARQUET.addExtension(UUID.randomUUID().toString()));
+ FileAppender<GenericData.Record> writer =
+ Parquet.write(localOutput(parquetFile)).schema(table.schema()).build();
+ try {
+ writer.addAll(expected);
+ } finally {
+ writer.close();
+ }
+
+ DataFile file =
+ DataFiles.builder(FAKE_SPEC)
+ .withInputFile(localInput(parquetFile))
+ .withMetrics(writer.metrics())
+ .withPartitionPath("id_zero=0")
+ .build();
+
+ OutputFile manifestFile = localOutput(FileFormat.AVRO.addExtension(temp.newFile().toString()));
+ ManifestWriter<DataFile> manifestWriter = ManifestFiles.write(FAKE_SPEC, manifestFile);
+ try {
+ manifestWriter.add(file);
+ } finally {
+ manifestWriter.close();
+ }
+
+ table.newFastAppend().appendManifest(manifestWriter.toManifestFile()).commit();
+
+ Dataset<Row> df = spark.read().format("iceberg").load(location.toString());
+
+ List<Row> rows = df.collectAsList();
+
+ Assert.assertEquals("Should contain 100 rows", 100, rows.size());
+
+ for (int i = 0; i < expected.size(); i += 1) {
+ TestHelpers.assertEqualsSafe(table.schema().asStruct(), expected.get(i), rows.get(i));
+ }
+ }
+
+ private <T> MemoryStream<T> newMemoryStream(int id, SQLContext sqlContext, Encoder<T> encoder) {
+ return new MemoryStream<>(id, sqlContext, Option.empty(), encoder);
+ }
+
+ private <T> void send(List<T> records, MemoryStream<T> stream) {
+ stream.addData(JavaConverters.asScalaBuffer(records));
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataFrameWriterV2.java
similarity index 81%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataFrameWriterV2.java
index 632c806075..b66015515d 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataFrameWriterV2.java
@@ -14,13 +14,8 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.spark.source.TestDataFrameWriterV2;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenDataFrameWriterV2 extends TestDataFrameWriterV2 {}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataFrameWriterV2Coercion.java
similarity index 69%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataFrameWriterV2Coercion.java
index 632c806075..f40b98bf18 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataFrameWriterV2Coercion.java
@@ -14,13 +14,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.spark.source.TestDataFrameWriterV2Coercion;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenDataFrameWriterV2Coercion extends TestDataFrameWriterV2Coercion {
+ public TestGlutenDataFrameWriterV2Coercion(FileFormat format, String dataType) {
+ super(format, dataType);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataSourceOptions.java
similarity index 81%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataSourceOptions.java
index 632c806075..11c2ecfe71 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenDataSourceOptions.java
@@ -14,13 +14,8 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.spark.source.TestDataSourceOptions;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenDataSourceOptions extends TestDataSourceOptions {}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenIcebergSourceHiveTables.java
similarity index 74%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenIcebergSourceHiveTables.java
index 632c806075..c3e921e324 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenIcebergSourceHiveTables.java
@@ -14,13 +14,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.spark.source.TestIcebergSourceHiveTables;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+// Fallback all the table scan because source table is metadata table with format avro.
+public class TestGlutenIcebergSourceHiveTables extends TestIcebergSourceHiveTables {}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenIdentityPartitionData.java
similarity index 67%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenIdentityPartitionData.java
index 632c806075..506f8a5226 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenIdentityPartitionData.java
@@ -14,13 +14,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.spark.source.TestIdentityPartitionData;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenIdentityPartitionData extends TestIdentityPartitionData {
+ public TestGlutenIdentityPartitionData(
+ String format, boolean vectorized, PlanningMode planningMode) {
+ super(format, vectorized, planningMode);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenPositionDeletesTable.java
similarity index 64%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenPositionDeletesTable.java
index 632c806075..02d348544d 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenPositionDeletesTable.java
@@ -14,13 +14,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.spark.source.TestPositionDeletesTable;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+import java.util.Map;
+
+public class TestGlutenPositionDeletesTable extends TestPositionDeletesTable {
+ public TestGlutenPositionDeletesTable(
+ String catalogName, String implementation, Map<String, String> config, FileFormat format) {
+ super(catalogName, implementation, config, format);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenRuntimeFiltering.java
similarity index 72%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenRuntimeFiltering.java
index 632c806075..90e3828991 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenRuntimeFiltering.java
@@ -14,13 +14,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.PlanningMode;
+import org.apache.iceberg.spark.source.TestRuntimeFiltering;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenRuntimeFiltering extends TestRuntimeFiltering {
+ public TestGlutenRuntimeFiltering(PlanningMode planningMode) {
+ super(planningMode);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenSparkMetadataColumns.java
similarity index 67%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenSparkMetadataColumns.java
index 632c806075..8e49b5876b 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenSparkMetadataColumns.java
@@ -14,13 +14,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.spark.source.TestSparkMetadataColumns;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+public class TestGlutenSparkMetadataColumns extends TestSparkMetadataColumns {
+ public TestGlutenSparkMetadataColumns(
+ FileFormat fileFormat, boolean vectorized, int formatVersion) {
+ super(fileFormat, vectorized, formatVersion);
+ }
+}
diff --git a/cpp/core/utils/Likely.h b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenSparkStagedScan.java
similarity index 69%
copy from cpp/core/utils/Likely.h
copy to backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenSparkStagedScan.java
index 632c806075..09a6583320 100644
--- a/cpp/core/utils/Likely.h
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestGlutenSparkStagedScan.java
@@ -14,13 +14,15 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+package org.apache.gluten.source;
-#pragma once
+import org.apache.iceberg.spark.source.TestSparkStagedScan;
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
\ No newline at end of file
+import java.util.Map;
+
+public class TestGlutenSparkStagedScan extends TestSparkStagedScan {
+ public TestGlutenSparkStagedScan(
+ String catalogName, String implementation, Map<String, String> config) {
+ super(catalogName, implementation, config);
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestIcebergSpark.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestIcebergSpark.java
new file mode 100644
index 0000000000..5a4d1079bf
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestIcebergSpark.java
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.source;
+
+import org.apache.gluten.TestConfUtil;
+
+import org.apache.iceberg.spark.IcebergSpark;
+import org.apache.iceberg.transforms.Transforms;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
+import org.apache.spark.sql.types.CharType;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.VarcharType;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.util.List;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+public class TestIcebergSpark {
+
+ private static SparkSession spark = null;
+
+ @BeforeClass
+ public static void startSpark() {
+ TestIcebergSpark.spark =
+ SparkSession.builder().master("local[2]").config(TestConfUtil.GLUTEN_CONF).getOrCreate();
+ }
+
+ @AfterClass
+ public static void stopSpark() {
+ SparkSession currentSpark = TestIcebergSpark.spark;
+ TestIcebergSpark.spark = null;
+ currentSpark.stop();
+ }
+
+ @Test
+ public void testRegisterIntegerBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_int_16", DataTypes.IntegerType, 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_int_16(1)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.IntegerType.get()).apply(1),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterShortBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_short_16", DataTypes.ShortType, 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_short_16(1S)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.IntegerType.get()).apply(1),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterByteBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_byte_16", DataTypes.ByteType, 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_byte_16(1Y)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.IntegerType.get()).apply(1),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterLongBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_long_16", DataTypes.LongType, 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_long_16(1L)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.LongType.get()).apply(1L), results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterStringBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_string_16", DataTypes.StringType, 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_string_16('hello')").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.StringType.get()).apply("hello"),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterCharBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_char_16", new CharType(5), 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_char_16('hello')").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.StringType.get()).apply("hello"),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterVarCharBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_varchar_16", new VarcharType(5), 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_varchar_16('hello')").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.StringType.get()).apply("hello"),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterDateBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_date_16", DataTypes.DateType, 16);
+ List<Row> results =
+ spark.sql("SELECT iceberg_bucket_date_16(DATE '2021-06-30')").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int)
+ Transforms.bucket(16)
+ .bind(Types.DateType.get())
+ .apply(DateTimeUtils.fromJavaDate(Date.valueOf("2021-06-30"))),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterTimestampBucketUDF() {
+ IcebergSpark.registerBucketUDF(
+ spark, "iceberg_bucket_timestamp_16", DataTypes.TimestampType, 16);
+ List<Row> results =
+ spark
+ .sql("SELECT iceberg_bucket_timestamp_16(TIMESTAMP '2021-06-30 00:00:00.000')")
+ .collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int)
+ Transforms.bucket(16)
+ .bind(Types.TimestampType.withZone())
+ .apply(
+ DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf("2021-06-30 00:00:00.000"))),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterBinaryBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_binary_16", DataTypes.BinaryType, 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_binary_16(X'0020001F')").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int)
+ Transforms.bucket(16)
+ .bind(Types.BinaryType.get())
+ .apply(ByteBuffer.wrap(new byte[] {0x00, 0x20, 0x00, 0x1F})),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterDecimalBucketUDF() {
+ IcebergSpark.registerBucketUDF(spark, "iceberg_bucket_decimal_16", new DecimalType(4, 2), 16);
+ List<Row> results = spark.sql("SELECT iceberg_bucket_decimal_16(11.11)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ (int) Transforms.bucket(16).bind(Types.DecimalType.of(4, 2)).apply(new BigDecimal("11.11")),
+ results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterBooleanBucketUDF() {
+ assertThatThrownBy(
+ () ->
+ IcebergSpark.registerBucketUDF(
+ spark, "iceberg_bucket_boolean_16", DataTypes.BooleanType, 16))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Cannot bucket by type: boolean");
+ }
+
+ @Test
+ public void testRegisterDoubleBucketUDF() {
+ assertThatThrownBy(
+ () ->
+ IcebergSpark.registerBucketUDF(
+ spark, "iceberg_bucket_double_16", DataTypes.DoubleType, 16))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Cannot bucket by type: double");
+ }
+
+ @Test
+ public void testRegisterFloatBucketUDF() {
+ assertThatThrownBy(
+ () ->
+ IcebergSpark.registerBucketUDF(
+ spark, "iceberg_bucket_float_16", DataTypes.FloatType, 16))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessage("Cannot bucket by type: float");
+ }
+
+ @Test
+ public void testRegisterIntegerTruncateUDF() {
+ IcebergSpark.registerTruncateUDF(spark, "iceberg_truncate_int_4", DataTypes.IntegerType, 4);
+ List<Row> results = spark.sql("SELECT iceberg_truncate_int_4(1)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ Transforms.truncate(4).bind(Types.IntegerType.get()).apply(1), results.get(0).getInt(0));
+ }
+
+ @Test
+ public void testRegisterLongTruncateUDF() {
+ IcebergSpark.registerTruncateUDF(spark, "iceberg_truncate_long_4", DataTypes.LongType, 4);
+ List<Row> results = spark.sql("SELECT iceberg_truncate_long_4(1L)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ Transforms.truncate(4).bind(Types.LongType.get()).apply(1L), results.get(0).getLong(0));
+ }
+
+ @Test
+ public void testRegisterDecimalTruncateUDF() {
+ IcebergSpark.registerTruncateUDF(spark, "iceberg_truncate_decimal_4", new DecimalType(4, 2), 4);
+ List<Row> results = spark.sql("SELECT iceberg_truncate_decimal_4(11.11)").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ Transforms.truncate(4).bind(Types.DecimalType.of(4, 2)).apply(new BigDecimal("11.11")),
+ results.get(0).getDecimal(0));
+ }
+
+ @Test
+ public void testRegisterStringTruncateUDF() {
+ IcebergSpark.registerTruncateUDF(spark, "iceberg_truncate_string_4", DataTypes.StringType, 4);
+ List<Row> results = spark.sql("SELECT iceberg_truncate_string_4('hello')").collectAsList();
+ Assert.assertEquals(1, results.size());
+ Assert.assertEquals(
+ Transforms.truncate(4).bind(Types.StringType.get()).apply("hello"),
+ results.get(0).getString(0));
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestParquetScan.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestParquetScan.java
new file mode 100644
index 0000000000..ef1e3becc0
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestParquetScan.java
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.source;
+
+import org.apache.gluten.TestConfUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.*;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.shaded.org.apache.avro.generic.GenericData;
+import org.apache.iceberg.spark.data.AvroDataTest;
+import org.apache.iceberg.spark.data.RandomData;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+import org.junit.*;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.iceberg.Files.localOutput;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.apache.spark.sql.functions.monotonically_increasing_id;
+import static org.assertj.core.api.Assertions.assertThat;
+
+// UUID and FixedType is not supported.
+@RunWith(Parameterized.class)
+public class TestParquetScan extends AvroDataTest {
+ private static final Configuration CONF = new Configuration();
+
+ private static SparkSession spark = null;
+
+ protected static final Types.StructType GLUTEN_SUPPORTED_PRIMITIVES =
+ Types.StructType.of(
+ required(100, "id", Types.LongType.get()),
+ optional(101, "data", Types.StringType.get()),
+ required(102, "b", Types.BooleanType.get()),
+ optional(103, "i", Types.IntegerType.get()),
+ required(104, "l", Types.LongType.get()),
+ optional(105, "f", Types.FloatType.get()),
+ required(106, "d", Types.DoubleType.get()),
+ optional(107, "date", Types.DateType.get()),
+ required(108, "ts", Types.TimestampType.withZone()),
+ required(110, "s", Types.StringType.get()),
+ optional(113, "bytes", Types.BinaryType.get()),
+ required(114, "dec_9_0", Types.DecimalType.of(9, 0)), // int encoded
+ required(115, "dec_11_2", Types.DecimalType.of(11, 2)), // long encoded
+ required(116, "dec_20_5", Types.DecimalType.of(20, 5)), // requires padding
+ required(117, "dec_38_10", Types.DecimalType.of(38, 10)) // Spark's maximum precision
+ );
+
+ @BeforeClass
+ public static void startSpark() {
+ TestParquetScan.spark =
+ SparkSession.builder().master("local[2]").config(TestConfUtil.GLUTEN_CONF).getOrCreate();
+ }
+
+ @AfterClass
+ public static void stopSpark() {
+ SparkSession currentSpark = TestParquetScan.spark;
+ TestParquetScan.spark = null;
+ currentSpark.stop();
+ }
+
+ @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+ @Parameterized.Parameters(name = "vectorized = {0}")
+ public static Object[] parameters() {
+ return new Object[] {false, true};
+ }
+
+ private final boolean vectorized;
+
+ public TestParquetScan(boolean vectorized) {
+ this.vectorized = vectorized;
+ }
+
+ @Override
+ protected void writeAndValidate(Schema schema) throws IOException {
+ Assume.assumeTrue(
+ "Cannot handle non-string map keys in parquet-avro",
+ null
+ == TypeUtil.find(
+ schema,
+ type -> type.isMapType() && type.asMapType().keyType() != Types.StringType.get()));
+
+ Table table = createTable(schema);
+
+ // Important: use the table's schema for the rest of the test
+ // When tables are created, the column ids are reassigned.
+ List<GenericData.Record> expected = RandomData.generateList(table.schema(), 100, 1L);
+ writeRecords(table, expected);
+
+ configureVectorization(table);
+ Dataset<Row> df = spark.read().format("iceberg").load(table.location());
+ List<Row> rows = df.collectAsList();
+ spark.conf().set("spark.gluten.enabled", "false");
+ List<Row> rows2 = df.collectAsList();
+
+ Assert.assertEquals("Should contain 100 rows", 100, rows.size());
+ assertThat(rows).containsExactlyInAnyOrderElementsOf(Iterables.concat(rows2));
+
+ spark.conf().set("spark.gluten.enabled", "true");
+ // Cannot use this helper test function because the order is not same.
+ // for (int i = 0; i < expected.size(); i += 1) {
+ // TestHelpers.assertEqualsSafe(table.schema().asStruct(), expected.get(i), rows.get(i));
+ // }
+ }
+
+ @Test
+ public void testEmptyTableProjection() throws IOException {
+ Types.StructType structType =
+ Types.StructType.of(
+ required(100, "id", Types.LongType.get()),
+ optional(101, "data", Types.StringType.get()),
+ required(102, "b", Types.BooleanType.get()),
+ optional(103, "i", Types.IntegerType.get()));
+ Table table = createTable(new Schema(structType.fields()));
+
+ List<GenericData.Record> expected = RandomData.generateList(table.schema(), 100, 1L);
+ writeRecords(table, expected);
+
+ configureVectorization(table);
+
+ List<Row> rows =
+ spark
+ .read()
+ .format("iceberg")
+ .load(table.location())
+ .select(monotonically_increasing_id())
+ .collectAsList();
+ assertThat(rows).hasSize(100);
+ }
+
+ @Test
+ public void testGlutenSimpleStruct() throws IOException {
+ this.writeAndValidate(
+ TypeUtil.assignIncreasingFreshIds(new Schema(GLUTEN_SUPPORTED_PRIMITIVES.fields())));
+ }
+
+ @Test
+ public void testSimpleStruct() throws IOException {
+ this.writeAndValidate(
+ TypeUtil.assignIncreasingFreshIds(new Schema(SUPPORTED_PRIMITIVES.fields())));
+ }
+
+ @Test
+ public void testArray() throws IOException {
+ Schema schema =
+ new Schema(
+ new Types.NestedField[] {
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(
+ 1, "data", Types.ListType.ofOptional(2, Types.StringType.get()))
+ });
+ this.writeAndValidate(schema);
+ }
+
+ // The result is right but order is not same.
+ // Use @Test to overwrite the super class test
+ @Test
+ public void testMap() throws IOException {}
+
+ // The result is right but order is not same.
+ // Use @Test to overwrite the super class test
+ @Test
+ public void testMapOfStructs() throws IOException {}
+
+ @Test
+ public void testMixedTypes() throws IOException {
+ Types.StructType structType =
+ Types.StructType.of(
+ required(0, "id", Types.LongType.get()),
+ optional(
+ 1,
+ "list_of_maps",
+ Types.ListType.ofOptional(
+ 2,
+ Types.MapType.ofOptional(
+ 3, 4, Types.StringType.get(), GLUTEN_SUPPORTED_PRIMITIVES))),
+ optional(
+ 5,
+ "map_of_lists",
+ Types.MapType.ofOptional(
+ 6,
+ 7,
+ Types.StringType.get(),
+ Types.ListType.ofOptional(8, GLUTEN_SUPPORTED_PRIMITIVES))),
+ required(
+ 9,
+ "list_of_lists",
+ Types.ListType.ofOptional(
+ 10, Types.ListType.ofOptional(11, GLUTEN_SUPPORTED_PRIMITIVES))),
+ required(
+ 12,
+ "map_of_maps",
+ Types.MapType.ofOptional(
+ 13,
+ 14,
+ Types.StringType.get(),
+ Types.MapType.ofOptional(
+ 15, 16, Types.StringType.get(), GLUTEN_SUPPORTED_PRIMITIVES))),
+ required(
+ 17,
+ "list_of_struct_of_nested_types",
+ Types.ListType.ofOptional(
+ 19,
+ Types.StructType.of(
+ Types.NestedField.required(
+ 20,
+ "m1",
+ Types.MapType.ofOptional(
+ 21, 22, Types.StringType.get(), GLUTEN_SUPPORTED_PRIMITIVES)),
+ Types.NestedField.optional(
+ 23, "l1", Types.ListType.ofRequired(24, GLUTEN_SUPPORTED_PRIMITIVES)),
+ Types.NestedField.required(
+ 25, "l2", Types.ListType.ofRequired(26, GLUTEN_SUPPORTED_PRIMITIVES)),
+ Types.NestedField.optional(
+ 27,
+ "m2",
+ Types.MapType.ofOptional(
+ 28, 29, Types.StringType.get(), GLUTEN_SUPPORTED_PRIMITIVES))))));
+
+ Schema schema =
+ new Schema(
+ TypeUtil.assignFreshIds(structType, new AtomicInteger(0)::incrementAndGet)
+ .asStructType()
+ .fields());
+
+ writeAndValidate(schema);
+ }
+
+ @Test
+ public void testStructWithOptionalFields() throws IOException {
+ this.writeAndValidate(
+ TypeUtil.assignIncreasingFreshIds(
+ new Schema(
+ Lists.transform(
+ GLUTEN_SUPPORTED_PRIMITIVES.fields(), Types.NestedField::asOptional))));
+ }
+
+ @Test
+ public void testStructWithRequiredFields() throws IOException {
+ this.writeAndValidate(
+ TypeUtil.assignIncreasingFreshIds(
+ new Schema(
+ Lists.transform(
+ GLUTEN_SUPPORTED_PRIMITIVES.fields(), Types.NestedField::asRequired))));
+ }
+
+ @Test
+ public void testArrayOfStructs() throws IOException {
+ Schema schema =
+ TypeUtil.assignIncreasingFreshIds(
+ new Schema(
+ new Types.NestedField[] {
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(
+ 1, "data", Types.ListType.ofOptional(2, GLUTEN_SUPPORTED_PRIMITIVES))
+ }));
+ this.writeAndValidate(schema);
+ }
+
+ @Test
+ public void testNestedStruct() throws IOException {
+ this.writeAndValidate(
+ TypeUtil.assignIncreasingFreshIds(
+ new Schema(
+ new Types.NestedField[] {
+ Types.NestedField.required(1, "struct", GLUTEN_SUPPORTED_PRIMITIVES)
+ })));
+ }
+
+ private Table createTable(Schema schema) throws IOException {
+ File parent = temp.newFolder("parquet");
+ File location = new File(parent, "test");
+ HadoopTables tables = new HadoopTables(CONF);
+ return tables.create(schema, PartitionSpec.unpartitioned(), location.toString());
+ }
+
+ private void writeRecords(Table table, List<GenericData.Record> records) throws IOException {
+ File dataFolder = new File(table.location(), "data");
+ dataFolder.mkdirs();
+
+ File parquetFile =
+ new File(dataFolder, FileFormat.PARQUET.addExtension(UUID.randomUUID().toString()));
+
+ try (FileAppender<GenericData.Record> writer =
+ Parquet.write(localOutput(parquetFile)).schema(table.schema()).build()) {
+ writer.addAll(records);
+ }
+
+ DataFile file =
+ DataFiles.builder(PartitionSpec.unpartitioned())
+ .withFileSizeInBytes(parquetFile.length())
+ .withPath(parquetFile.toString())
+ .withRecordCount(100)
+ .build();
+
+ table.newAppend().appendFile(file).commit();
+ }
+
+ private void configureVectorization(Table table) {
+ table
+ .updateProperties()
+ .set(TableProperties.PARQUET_VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .commit();
+ }
+}
diff --git a/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestPartitionPruning.java b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestPartitionPruning.java
new file mode 100644
index 0000000000..ceb5799db7
--- /dev/null
+++ b/backends-bolt/src-iceberg-spark34/test/java/org/apache/gluten/source/TestPartitionPruning.java
@@ -0,0 +1,479 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.source;
+
+import org.apache.gluten.TestConfUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RawLocalFileSystem;
+import org.apache.iceberg.*;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.spark.SparkReadOptions;
+import org.apache.iceberg.spark.SparkSchemaUtil;
+import org.apache.iceberg.spark.source.LogMessage;
+import org.apache.iceberg.transforms.Transforms;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.*;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static org.apache.iceberg.PlanningMode.DISTRIBUTED;
+import static org.apache.iceberg.PlanningMode.LOCAL;
+
+// TODO: remove the file format parquet to make the tests pass, file system match issue
+@RunWith(Parameterized.class)
+public class TestPartitionPruning {
+
+ private static final Configuration CONF = new Configuration();
+ private static final HadoopTables TABLES = new HadoopTables(CONF);
+
+ @Parameterized.Parameters(name = "format = {0}, vectorized = {1}, planningMode = {2}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ // {"parquet", false, DISTRIBUTED},
+ // {"parquet", true, LOCAL},
+ {"avro", false, DISTRIBUTED},
+ {"orc", false, LOCAL},
+ {"orc", true, DISTRIBUTED}
+ };
+ }
+
+ private final String format;
+ private final boolean vectorized;
+ private final PlanningMode planningMode;
+
+ public TestPartitionPruning(String format, boolean vectorized, PlanningMode planningMode) {
+ this.format = format;
+ this.vectorized = vectorized;
+ this.planningMode = planningMode;
+ }
+
+ private static SparkSession spark = null;
+ private static JavaSparkContext sparkContext = null;
+
+ private static final Function<Object, Integer> BUCKET_FUNC =
+ Transforms.bucket(3).bind(Types.IntegerType.get());
+ private static final Function<Object, Object> TRUNCATE_FUNC =
+ Transforms.truncate(5).bind(Types.StringType.get());
+ private static final Function<Object, Integer> HOUR_FUNC =
+ Transforms.hour().bind(Types.TimestampType.withoutZone());
+
+ @BeforeClass
+ public static void startSpark() {
+ TestPartitionPruning.spark =
+ SparkSession.builder().master("local[2]").config(TestConfUtil.GLUTEN_CONF).getOrCreate();
+ TestPartitionPruning.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
+
+ String optionKey = String.format("fs.%s.impl", CountOpenLocalFileSystem.scheme);
+ CONF.set(optionKey, CountOpenLocalFileSystem.class.getName());
+ spark.conf().set(optionKey, CountOpenLocalFileSystem.class.getName());
+ spark.conf().set("spark.sql.session.timeZone", "UTC");
+ spark.udf().register("bucket3", (Integer num) -> BUCKET_FUNC.apply(num), DataTypes.IntegerType);
+ spark
+ .udf()
+ .register("truncate5", (String str) -> TRUNCATE_FUNC.apply(str), DataTypes.StringType);
+ // NOTE: date transforms take the type long, not Timestamp
+ spark
+ .udf()
+ .register(
+ "hour",
+ (Timestamp ts) ->
+ HOUR_FUNC.apply(
+ org.apache.spark.sql.catalyst.util.DateTimeUtils.fromJavaTimestamp(ts)),
+ DataTypes.IntegerType);
+ }
+
+ @AfterClass
+ public static void stopSpark() {
+ SparkSession currentSpark = TestPartitionPruning.spark;
+ TestPartitionPruning.spark = null;
+ currentSpark.stop();
+ }
+
+ private static final Schema LOG_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.IntegerType.get()),
+ Types.NestedField.optional(2, "date", Types.StringType.get()),
+ Types.NestedField.optional(3, "level", Types.StringType.get()),
+ Types.NestedField.optional(4, "message", Types.StringType.get()),
+ Types.NestedField.optional(5, "timestamp", Types.TimestampType.withZone()));
+
+ private static final List<LogMessage> LOGS =
+ ImmutableList.of(
+ LogMessage.debug("2020-02-02", "debug event 1", getInstant("2020-02-02T00:00:00")),
+ LogMessage.info("2020-02-02", "info event 1", getInstant("2020-02-02T01:00:00")),
+ LogMessage.debug("2020-02-02", "debug event 2", getInstant("2020-02-02T02:00:00")),
+ LogMessage.info("2020-02-03", "info event 2", getInstant("2020-02-03T00:00:00")),
+ LogMessage.debug("2020-02-03", "debug event 3", getInstant("2020-02-03T01:00:00")),
+ LogMessage.info("2020-02-03", "info event 3", getInstant("2020-02-03T02:00:00")),
+ LogMessage.error("2020-02-03", "error event 1", getInstant("2020-02-03T03:00:00")),
+ LogMessage.debug("2020-02-04", "debug event 4", getInstant("2020-02-04T01:00:00")),
+ LogMessage.warn("2020-02-04", "warn event 1", getInstant("2020-02-04T02:00:00")),
+ LogMessage.debug("2020-02-04", "debug event 5", getInstant("2020-02-04T03:00:00")));
+
+ private static Instant getInstant(String timestampWithoutZone) {
+ Long epochMicros =
+ (Long) Literal.of(timestampWithoutZone).to(Types.TimestampType.withoutZone()).value();
+ return Instant.ofEpochMilli(TimeUnit.MICROSECONDS.toMillis(epochMicros));
+ }
+
+ @Rule public TemporaryFolder temp = new TemporaryFolder();
+
+ private final PartitionSpec spec =
+ PartitionSpec.builderFor(LOG_SCHEMA)
+ .identity("date")
+ .identity("level")
+ .bucket("id", 3)
+ .truncate("message", 5)
+ .hour("timestamp")
+ .build();
+
+ @Test
+ public void testPartitionPruningIdentityString() {
+ String filterCond = "date >= '2020-02-03' AND level = 'DEBUG'";
+ Predicate<Row> partCondition =
+ (Row r) -> {
+ String date = r.getString(0);
+ String level = r.getString(1);
+ return date.compareTo("2020-02-03") >= 0 && level.equals("DEBUG");
+ };
+
+ runTest(filterCond, partCondition);
+ }
+
+ @Test
+ public void testPartitionPruningBucketingInteger() {
+ final int[] ids = new int[] {LOGS.get(3).getId(), LOGS.get(7).getId()};
+ String condForIds =
+ Arrays.stream(ids).mapToObj(String::valueOf).collect(Collectors.joining(",", "(", ")"));
+ String filterCond = "id in " + condForIds;
+ Predicate<Row> partCondition =
+ (Row r) -> {
+ int bucketId = r.getInt(2);
+ Set<Integer> buckets =
+ Arrays.stream(ids).map(BUCKET_FUNC::apply).boxed().collect(Collectors.toSet());
+ return buckets.contains(bucketId);
+ };
+
+ runTest(filterCond, partCondition);
+ }
+
+ @Test
+ public void testPartitionPruningTruncatedString() {
+ String filterCond = "message like 'info event%'";
+ Predicate<Row> partCondition =
+ (Row r) -> {
+ String truncatedMessage = r.getString(3);
+ return truncatedMessage.equals("info ");
+ };
+
+ runTest(filterCond, partCondition);
+ }
+
+ @Test
+ public void testPartitionPruningTruncatedStringComparingValueShorterThanPartitionValue() {
+ String filterCond = "message like 'inf%'";
+ Predicate<Row> partCondition =
+ (Row r) -> {
+ String truncatedMessage = r.getString(3);
+ return truncatedMessage.startsWith("inf");
+ };
+
+ runTest(filterCond, partCondition);
+ }
+
+ @Test
+ public void testPartitionPruningHourlyPartition() {
+ String filterCond;
+ if (spark.version().startsWith("2")) {
+ // Looks like from Spark 2 we need to compare timestamp with timestamp to push down the
+ // filter.
+ filterCond = "timestamp >= to_timestamp('2020-02-03T01:00:00')";
+ } else {
+ filterCond = "timestamp >= '2020-02-03T01:00:00'";
+ }
+ Predicate<Row> partCondition =
+ (Row r) -> {
+ int hourValue = r.getInt(4);
+ Instant instant = getInstant("2020-02-03T01:00:00");
+ Integer hourValueToFilter =
+ HOUR_FUNC.apply(TimeUnit.MILLISECONDS.toMicros(instant.toEpochMilli()));
+ return hourValue >= hourValueToFilter;
+ };
+
+ runTest(filterCond, partCondition);
+ }
+
+ private void runTest(String filterCond, Predicate<Row> partCondition) {
+ File originTableLocation = createTempDir();
+ Assert.assertTrue("Temp folder should exist", originTableLocation.exists());
+
+ Table table = createTable(originTableLocation);
+ Dataset<Row> logs = createTestDataset();
+ saveTestDatasetToTable(logs, table);
+
+ List<Row> expected =
+ logs.select("id", "date", "level", "message", "timestamp")
+ .filter(filterCond)
+ .orderBy("id")
+ .collectAsList();
+ Assert.assertFalse("Expected rows should be not empty", expected.isEmpty());
+
+ // remove records which may be recorded during storing to table
+ CountOpenLocalFileSystem.resetRecordsInPathPrefix(originTableLocation.getAbsolutePath());
+
+ List<Row> actual =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(table.location())
+ .select("id", "date", "level", "message", "timestamp")
+ .filter(filterCond)
+ .orderBy("id")
+ .collectAsList();
+
+ Dataset<Row> ros =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(table.location())
+ .select("id", "date", "level", "message", "timestamp")
+ .filter(filterCond)
+ .orderBy("id");
+ ros.collectAsList();
+ System.out.println(ros.queryExecution().executedPlan());
+ Assert.assertFalse("Actual rows should not be empty", actual.isEmpty());
+
+ Assert.assertEquals("Rows should match", expected, actual);
+
+ assertAccessOnDataFiles(originTableLocation, table, partCondition);
+ }
+
+ private File createTempDir() {
+ try {
+ return temp.newFolder();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private Table createTable(File originTableLocation) {
+ String trackedTableLocation = CountOpenLocalFileSystem.convertPath(originTableLocation);
+ Map<String, String> properties =
+ ImmutableMap.of(
+ TableProperties.DEFAULT_FILE_FORMAT, format,
+ TableProperties.DATA_PLANNING_MODE, planningMode.modeName(),
+ TableProperties.DELETE_PLANNING_MODE, planningMode.modeName());
+ return TABLES.create(LOG_SCHEMA, spec, properties, trackedTableLocation);
+ }
+
+ private Dataset<Row> createTestDataset() {
+ List<InternalRow> rows =
+ LOGS.stream()
+ .map(
+ logMessage -> {
+ Object[] underlying =
+ new Object[] {
+ logMessage.getId(),
+ UTF8String.fromString(logMessage.getDate()),
+ UTF8String.fromString(logMessage.getLevel()),
+ UTF8String.fromString(logMessage.getMessage()),
+ // discard the nanoseconds part to simplify
+ TimeUnit.MILLISECONDS.toMicros(logMessage.getTimestamp().toEpochMilli())
+ };
+ return new GenericInternalRow(underlying);
+ })
+ .collect(Collectors.toList());
+
+ JavaRDD<InternalRow> rdd = sparkContext.parallelize(rows);
+ Dataset<Row> df =
+ spark.internalCreateDataFrame(
+ JavaRDD.toRDD(rdd), SparkSchemaUtil.convert(LOG_SCHEMA), false);
+
+ return df.selectExpr("id", "date", "level", "message", "timestamp")
+ .selectExpr(
+ "id",
+ "date",
+ "level",
+ "message",
+ "timestamp",
+ "bucket3(id) AS bucket_id",
+ "truncate5(message) AS truncated_message",
+ "hour(timestamp) AS ts_hour");
+ }
+
+ private void saveTestDatasetToTable(Dataset<Row> logs, Table table) {
+ logs.orderBy("date", "level", "bucket_id", "truncated_message", "ts_hour")
+ .select("id", "date", "level", "message", "timestamp")
+ .write()
+ .format("iceberg")
+ .mode("append")
+ .save(table.location());
+ }
+
+ private void assertAccessOnDataFiles(
+ File originTableLocation, Table table, Predicate<Row> partCondition) {
+ // only use files in current table location to avoid side-effects on concurrent test runs
+ Set<String> readFilesInQuery =
+ CountOpenLocalFileSystem.pathToNumOpenCalled.keySet().stream()
+ .filter(path -> path.startsWith(originTableLocation.getAbsolutePath()))
+ .collect(Collectors.toSet());
+
+ List<Row> files =
+ spark.read().format("iceberg").load(table.location() + "#files").collectAsList();
+
+ Set<String> filesToRead = extractFilePathsMatchingConditionOnPartition(files, partCondition);
+ Set<String> filesToNotRead = extractFilePathsNotIn(files, filesToRead);
+
+ // Just to be sure, they should be mutually exclusive.
+ Assert.assertTrue(Sets.intersection(filesToRead, filesToNotRead).isEmpty());
+
+ Assert.assertFalse("The query should prune some data files.", filesToNotRead.isEmpty());
+
+ // We don't check "all" data files bound to the condition are being read, as data files can be
+ // pruned on
+ // other conditions like lower/upper bound of columns.
+ Assert.assertFalse(
+ "Some of data files in partition range should be read. "
+ + "Read files in query: "
+ + readFilesInQuery
+ + " / data files in partition range: "
+ + filesToRead,
+ Sets.intersection(filesToRead, readFilesInQuery).isEmpty());
+
+ // Data files which aren't bound to the condition shouldn't be read.
+ Assert.assertTrue(
+ "Data files outside of partition range should not be read. "
+ + "Read files in query: "
+ + readFilesInQuery
+ + " / data files outside of partition range: "
+ + filesToNotRead,
+ Sets.intersection(filesToNotRead, readFilesInQuery).isEmpty());
+ }
+
+ private Set<String> extractFilePathsMatchingConditionOnPartition(
+ List<Row> files, Predicate<Row> condition) {
+ // idx 1: file_path, idx 3: partition
+ return files.stream()
+ .filter(
+ r -> {
+ Row partition = r.getStruct(4);
+ return condition.test(partition);
+ })
+ .map(r -> CountOpenLocalFileSystem.stripScheme(r.getString(1)))
+ .collect(Collectors.toSet());
+ }
+
+ private Set<String> extractFilePathsNotIn(List<Row> files, Set<String> filePaths) {
+ Set<String> allFilePaths =
+ files.stream()
+ .map(r -> CountOpenLocalFileSystem.stripScheme(r.getString(1)))
+ .collect(Collectors.toSet());
+ return Sets.newHashSet(Sets.symmetricDifference(allFilePaths, filePaths));
+ }
+
+ public static class CountOpenLocalFileSystem extends RawLocalFileSystem {
+ public static String scheme =
+ String.format("TestIdentityPartitionData%dfs", new Random().nextInt());
+ public static Map<String, Long> pathToNumOpenCalled = Maps.newConcurrentMap();
+
+ public static String convertPath(String absPath) {
+ return scheme + "://" + absPath;
+ }
+
+ public static String convertPath(File file) {
+ return convertPath(file.getAbsolutePath());
+ }
+
+ public static String stripScheme(String pathWithScheme) {
+ if (!pathWithScheme.startsWith(scheme + ":")) {
+ throw new IllegalArgumentException("Received unexpected path: " + pathWithScheme);
+ }
+
+ int idxToCut = scheme.length() + 1;
+ while (pathWithScheme.charAt(idxToCut) == '/') {
+ idxToCut++;
+ }
+
+ // leave the last '/'
+ idxToCut--;
+
+ return pathWithScheme.substring(idxToCut);
+ }
+
+ public static void resetRecordsInPathPrefix(String pathPrefix) {
+ pathToNumOpenCalled.keySet().stream()
+ .filter(p -> p.startsWith(pathPrefix))
+ .forEach(key -> pathToNumOpenCalled.remove(key));
+ }
+
+ @Override
+ public URI getUri() {
+ return URI.create(scheme + ":///");
+ }
+
+ @Override
+ public String getScheme() {
+ return scheme;
+ }
+
+ @Override
+ public FSDataInputStream open(Path f, int bufferSize) throws IOException {
+ String path = f.toUri().getPath();
+ pathToNumOpenCalled.compute(
+ path,
+ (ignored, v) -> {
+ if (v == null) {
+ return 1L;
+ } else {
+ return v + 1;
+ }
+ });
+ return super.open(f, bufferSize);
... 234358 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 37/38: [GLUTEN][CORE] Share additional tier-2 test suites
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit ee2f50bad6f033894867f52bebff17bbf91194a9
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Wed Jul 1 15:39:27 2026 +0800
[GLUTEN][CORE] Share additional tier-2 test suites
Move ArrowEvalPythonExecSuite and UDFPartialProjectSuite into backends-core, and switch their plan assertions to reflective lookup so the shared tests no longer depend on backend-only test-time symbols.
---
.../python/ArrowEvalPythonExecSuite.scala | 29 ++++++++-
.../gluten/expression/UDFPartialProjectSuite.scala | 69 +++++++++++++++-------
2 files changed, 74 insertions(+), 24 deletions(-)
diff --git a/backends-velox/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala b/backends-core/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
similarity index 82%
rename from backends-velox/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
index 7747bd2193..808b82040b 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
+++ b/backends-core/src/test/scala/org/apache/gluten/execution/python/ArrowEvalPythonExecSuite.scala
@@ -19,13 +19,16 @@ package org.apache.gluten.execution.python
import org.apache.gluten.execution.WholeStageTransformerSuite
import org.apache.spark.SparkConf
-import org.apache.spark.api.python.ColumnarArrowEvalPythonExec
+import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.IntegratedUDFTestUtils
import org.apache.spark.sql.types.{DataType, LongType, StringType}
import org.apache.spark.util.SparkVersionUtil
class ArrowEvalPythonExecSuite extends WholeStageTransformerSuite {
+ private val columnarArrowEvalPythonExecClassName =
+ "org.apache.spark.api.python.ColumnarArrowEvalPythonExec"
+
import IntegratedUDFTestUtils._
import testImplicits.localSeqToDatasetHolder
import testImplicits.newProductEncoder
@@ -61,7 +64,7 @@ class ArrowEvalPythonExecSuite extends WholeStageTransformerSuite {
).toDF("a", "p_a")
val df2 = base.select("a").withColumn("p_a", pyarrowTestUDFString(base("a")))
- checkSparkPlan[ColumnarArrowEvalPythonExec](df2)
+ checkColumnarArrowEvalPythonExec(df2)
checkAnswer(df2, expected)
}
@@ -83,7 +86,7 @@ class ArrowEvalPythonExecSuite extends WholeStageTransformerSuite {
val df =
base.withColumn("p_a", pyarrowTestUDFString(base("a"))).withColumn("d_b", base("b") * 2)
- checkSparkPlan[ColumnarArrowEvalPythonExec](df)
+ checkColumnarArrowEvalPythonExec(df)
checkAnswer(df, expected)
}
@@ -123,4 +126,24 @@ class ArrowEvalPythonExecSuite extends WholeStageTransformerSuite {
TestScalarPandasUDF(name)
}
}
+
+ private def checkColumnarArrowEvalPythonExec(df: DataFrame): Unit = {
+ val planClass = loadPlanClass(columnarArrowEvalPythonExecClassName)
+ val executedPlan = getExecutedPlan(df)
+ assert(
+ executedPlan.exists(planClass.isInstance),
+ s"Expect ${planClass.getSimpleName} exists in executedPlan:\n ${executedPlan.last}"
+ )
+ }
+
+ private def loadPlanClass(className: String): Class[_] = {
+ try {
+ // scalastyle:off classforname
+ Class.forName(className)
+ // scalastyle:on classforname
+ } catch {
+ case e: ClassNotFoundException =>
+ fail(s"$className is not available on current backend classpath", e)
+ }
+ }
}
diff --git a/backends-velox/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala b/backends-core/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala
similarity index 83%
rename from backends-velox/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala
index 41cd9ac186..edb3808b2f 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala
+++ b/backends-core/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala
@@ -17,9 +17,10 @@
package org.apache.gluten.expression
import org.apache.gluten.config.GlutenConfig
-import org.apache.gluten.execution.{ColumnarPartialProjectExec, WholeStageTransformerSuite}
+import org.apache.gluten.execution.WholeStageTransformerSuite
import org.apache.spark.SparkConf
+import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.catalyst.optimizer.{ConstantFolding, NullPropagation}
import org.apache.spark.sql.execution.ProjectExec
import org.apache.spark.sql.functions.udf
@@ -31,6 +32,9 @@ case class MyStruct(a: Long, b: Array[Long])
case class MyStructWithNullValue(a: Option[Long], b: Array[Long])
class UDFPartialProjectSuite extends WholeStageTransformerSuite {
+ private val columnarPartialProjectExecClassName =
+ "org.apache.gluten.execution.ColumnarPartialProjectExec"
+
disableFallbackCheck
override protected val resourcePath: String = "/tpch-data-parquet"
override protected val fileFormat: String = "parquet"
@@ -69,7 +73,7 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
ignore("test plus_one") {
runQueryAndCompare("SELECT sum(plus_one(cast(l_orderkey as long))) from lineitem") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -78,20 +82,20 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
"select plus_one(" +
"(select plus_one(count(*)) from (values (1)) t0(inner_c))) as col " +
"from (values (2),(3)) t1(outer_c)") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
ignore("test plus_one with column used twice") {
runQueryAndCompare(
"SELECT sum(plus_one(cast(l_orderkey as long)) + hash(l_orderkey)) from lineitem") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
ignore("test plus_one without cast") {
runQueryAndCompare("SELECT sum(plus_one(l_orderkey) + hash(l_orderkey)) from lineitem") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -100,7 +104,7 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
"SELECT sum(plus_one(cast(l_orderkey as long)) + hash(l_partkey))" +
"from lineitem " +
"where l_orderkey < 3") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -110,25 +114,23 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
| select plus_one(l_orderkey) as col1, l_partkey from lineitem
|)""".stripMargin
runQueryAndCompare(sql) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
val df = spark.sql(sql)
- assert(df.queryExecution.executedPlan.collect {
- case p: ColumnarPartialProjectExec => p
- }.size == 2)
+ assert(countColumnarPartialProjectExec(df) == 2)
}
test("test plus_one with many columns in project") {
runQueryAndCompare("SELECT plus_one(cast(l_orderkey as long)), hash(l_partkey) from lineitem") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
ignore("test function no argument") {
runQueryAndCompare("""SELECT no_argument(), l_orderkey
| from lineitem limit 100""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -138,14 +140,14 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
df.collect()
assert(
df.queryExecution.executedPlan
- .find(p => p.isInstanceOf[ColumnarPartialProjectExec])
+ .find(loadPlanClass(columnarPartialProjectExecClassName).isInstance)
.isEmpty)
}
test("udf in agg simple") {
runQueryAndCompare("""select sum(hash(plus_one(l_extendedprice)) + hash(l_orderkey) ) as revenue
| from lineitem""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -153,13 +155,13 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
runQueryAndCompare("""select sum(hash(plus_one(l_extendedprice)) * l_discount
| + hash(l_orderkey) + hash(l_comment)) as revenue
| from lineitem""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
test("test concat with string") {
runQueryAndCompare("SELECT concat_concat(l_comment), hash(l_partkey) from lineitem") {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -175,7 +177,7 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
| GROUP BY l_partkey
|)
|""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -196,7 +198,7 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
| FROM lineitem
|)
|""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
@@ -215,7 +217,7 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
| FROM lineitem
|)
|""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
}
}
// only SparkVersion >= 3.4 support columnar native writer
@@ -241,7 +243,7 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
|""".stripMargin) {
if (enableNativeScanAndWriter.toBoolean) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
} else {
checkSparkPlan[ProjectExec]
}
@@ -270,7 +272,32 @@ class UDFPartialProjectSuite extends WholeStageTransformerSuite {
| FROM lineitem
|)
|""".stripMargin) {
- checkGlutenPlan[ColumnarPartialProjectExec]
+ checkColumnarPartialProjectExec
+ }
+ }
+
+ private def checkColumnarPartialProjectExec(df: DataFrame): Unit = {
+ val planClass = loadPlanClass(columnarPartialProjectExecClassName)
+ val executedPlan = getExecutedPlan(df)
+ assert(
+ executedPlan.exists(planClass.isInstance),
+ s"Expect ${planClass.getSimpleName} exists in executedPlan:\n ${executedPlan.last}"
+ )
+ }
+
+ private def countColumnarPartialProjectExec(df: DataFrame): Int = {
+ val planClass = loadPlanClass(columnarPartialProjectExecClassName)
+ getExecutedPlan(df).count(planClass.isInstance)
+ }
+
+ private def loadPlanClass(className: String): Class[_] = {
+ try {
+ // scalastyle:off classforname
+ Class.forName(className)
+ // scalastyle:on classforname
+ } catch {
+ case e: ClassNotFoundException =>
+ fail(s"$className is not available on current backend classpath", e)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 31/38: [GLUTEN][CORE] Switch shared RowToColumnarFuzzer to SharedRowToColumnarExec
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit e66e23e3655d7109a282a22bc53c9240932189ab
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 21:16:22 2026 +0800
[GLUTEN][CORE] Switch shared RowToColumnarFuzzer to SharedRowToColumnarExec
After the previous patch extracted SharedRowToColumnarExec as the
backend-agnostic base, the shared RowToColumnarFuzzer can now match on
SharedRowToColumnarExec instead of the velox-specific subclass. This
unbinds the fuzzer from any backend so all backends can reuse it
directly.
Generated-by: TraeCli openrouter-3o
---
.../src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/backends-core/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala b/backends-core/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala
index eeff98dccb..a74b8e2476 100644
--- a/backends-core/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala
+++ b/backends-core/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala
@@ -17,7 +17,7 @@
package org.apache.gluten.fuzzer
import org.apache.gluten.config.GlutenConfig
-import org.apache.gluten.execution.RowToVeloxColumnarExec
+import org.apache.gluten.execution.SharedRowToColumnarExec
import org.apache.gluten.fuzzer.FuzzerResult.Successful
import org.apache.gluten.tags.{FuzzerTest, SkipTest}
@@ -34,7 +34,7 @@ class RowToColumnarFuzzer extends FuzzerBase {
}
private def checkOperators(df: DataFrame): Unit = {
- checkGlutenPlan[RowToVeloxColumnarExec](df)
+ checkGlutenPlan[SharedRowToColumnarExec](df)
}
private val TEST_ROW_TO_COLUMNAR = "row to columnar"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 32/38: [GLUTEN][CORE] Share JNI wrapper classes under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 0302293fa4031d58d80dc15fbbc7dca23a386cbc
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 21:26:22 2026 +0800
[GLUTEN][CORE] Share JNI wrapper classes under backends-core
Lift five JNI wrapper classes that are byte-identical between Velox-
and Bolt-style backends (after normalising backend naming) into
backends-core so both sides can share them:
- org.apache.gluten.config.ConfigJniWrapper
- org.apache.gluten.metrics.IteratorMetricsJniWrapper
- org.apache.gluten.udf.UdfJniWrapper
- org.apache.gluten.fs.JniFilesystem
- org.apache.gluten.fs.OnHeapFileSystem
Add a compile-scope dependency on gluten-arrow to backends-core (only
needed for IteratorMetricsJniWrapper which references
ColumnarBatchOutIterator). No behaviour change: the fully-qualified
package names are preserved so native symbol lookups
(Java_org_apache_gluten_*) keep resolving unchanged.
---
backends-core/pom.xml | 6 ++++++
.../src/main/java/org/apache/gluten/config/ConfigJniWrapper.java | 0
.../src/main/java/org/apache/gluten/fs/JniFilesystem.java | 0
.../src/main/java/org/apache/gluten/fs/OnHeapFileSystem.java | 0
.../java/org/apache/gluten/metrics/IteratorMetricsJniWrapper.java | 0
.../src/main/java/org/apache/gluten/udf/UdfJniWrapper.java | 0
6 files changed, 6 insertions(+)
diff --git a/backends-core/pom.xml b/backends-core/pom.xml
index 22a445eb27..64dcfbe29e 100644
--- a/backends-core/pom.xml
+++ b/backends-core/pom.xml
@@ -35,6 +35,12 @@
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.gluten</groupId>
+ <artifactId>gluten-arrow</artifactId>
+ <version>${project.version}</version>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
diff --git a/backends-velox/src/main/java/org/apache/gluten/config/ConfigJniWrapper.java b/backends-core/src/main/java/org/apache/gluten/config/ConfigJniWrapper.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/config/ConfigJniWrapper.java
rename to backends-core/src/main/java/org/apache/gluten/config/ConfigJniWrapper.java
diff --git a/backends-velox/src/main/java/org/apache/gluten/fs/JniFilesystem.java b/backends-core/src/main/java/org/apache/gluten/fs/JniFilesystem.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/fs/JniFilesystem.java
rename to backends-core/src/main/java/org/apache/gluten/fs/JniFilesystem.java
diff --git a/backends-velox/src/main/java/org/apache/gluten/fs/OnHeapFileSystem.java b/backends-core/src/main/java/org/apache/gluten/fs/OnHeapFileSystem.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/fs/OnHeapFileSystem.java
rename to backends-core/src/main/java/org/apache/gluten/fs/OnHeapFileSystem.java
diff --git a/backends-velox/src/main/java/org/apache/gluten/metrics/IteratorMetricsJniWrapper.java b/backends-core/src/main/java/org/apache/gluten/metrics/IteratorMetricsJniWrapper.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/metrics/IteratorMetricsJniWrapper.java
rename to backends-core/src/main/java/org/apache/gluten/metrics/IteratorMetricsJniWrapper.java
diff --git a/backends-velox/src/main/java/org/apache/gluten/udf/UdfJniWrapper.java b/backends-core/src/main/java/org/apache/gluten/udf/UdfJniWrapper.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/udf/UdfJniWrapper.java
rename to backends-core/src/main/java/org/apache/gluten/udf/UdfJniWrapper.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 04/38: [GLUTEN][CORE] Defer literal node construction for InSet to reduce memory
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 28a3a9e733158abfefafbf53504a75ac79eb051e
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Fri Jun 26 17:52:04 2026 +0800
[GLUTEN][CORE] Defer literal node construction for InSet to reduce memory
Keep raw IN-set values and the value data type on SingularOrListNode and build
the substrait LiteralNode options lazily in toProtobuf(), instead of eagerly
materializing one LiteralNode per value. This avoids holding many LiteralNode
objects in memory for large IN lists.
- Add ExpressionBuilder.makeSingularOrListNode(value, rawValues, dataType) and a
matching SingularOrListNode constructor; build options from raw values lazily.
- InSetTransformer passes sorted raw hset values (deterministic) and defers
LiteralNode creation to protobuf serialization.
Generated-by: TraeCli openrouter-3o
---
.../substrait/expression/ExpressionBuilder.java | 5 ++++
.../substrait/expression/SingularOrListNode.java | 32 ++++++++++++++++++++--
.../PredicateExpressionTransformer.scala | 16 ++++++++---
3 files changed, 47 insertions(+), 6 deletions(-)
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/ExpressionBuilder.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/ExpressionBuilder.java
index 4bdef37878..547b4dd4c4 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/ExpressionBuilder.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/ExpressionBuilder.java
@@ -251,6 +251,11 @@ public class ExpressionBuilder {
return new SingularOrListNode(value, expressionNodes);
}
+ public static SingularOrListNode makeSingularOrListNode(
+ ExpressionNode value, List<Object> rawValues, org.apache.spark.sql.types.DataType dataType) {
+ return new SingularOrListNode(value, rawValues, dataType);
+ }
+
public static WindowFunctionNode makeWindowFunction(
Integer functionId,
List<ExpressionNode> expressionNodes,
diff --git a/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/SingularOrListNode.java b/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/SingularOrListNode.java
index c55791a085..6371abd676 100644
--- a/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/SingularOrListNode.java
+++ b/gluten-substrait/src/main/java/org/apache/gluten/substrait/expression/SingularOrListNode.java
@@ -17,6 +17,7 @@
package org.apache.gluten.substrait.expression;
import io.substrait.proto.Expression;
+import org.apache.spark.sql.types.DataType;
import java.io.Serializable;
import java.util.ArrayList;
@@ -25,19 +26,46 @@ import java.util.List;
public class SingularOrListNode implements ExpressionNode, Serializable {
private final ExpressionNode value;
private final List<ExpressionNode> listNodes = new ArrayList<>();
+ // rawValues and dataType allow delaying literal node construction until toProtobuf()
+ private final List<Object> rawValues;
+ private final DataType dataType;
SingularOrListNode(ExpressionNode value, List<ExpressionNode> listNodes) {
this.value = value;
this.listNodes.addAll(listNodes);
+ this.rawValues = null;
+ this.dataType = null;
+ }
+
+ SingularOrListNode(ExpressionNode value, List<Object> rawValues, DataType dataType) {
+ this.value = value;
+ this.rawValues = new ArrayList<>(rawValues);
+ this.dataType = dataType;
}
@Override
public Expression toProtobuf() {
Expression.SingularOrList.Builder builder = Expression.SingularOrList.newBuilder();
builder.setValue(value.toProtobuf());
- for (ExpressionNode expressionNode : listNodes) {
- builder.addOptions(expressionNode.toProtobuf());
+ List<Expression> options;
+ if (!listNodes.isEmpty()) {
+ options = new ArrayList<>(listNodes.size());
+ for (ExpressionNode expressionNode : listNodes) {
+ options.add(expressionNode.toProtobuf());
+ }
+ } else if (rawValues != null) {
+ options = new ArrayList<>(rawValues.size());
+ for (Object obj : rawValues) {
+ // construct a temporary LiteralNode and convert to protobuf to avoid keeping
+ // many LiteralNode objects in memory at once. Use per-value nullability.
+ LiteralNode literalNode =
+ (LiteralNode) ExpressionBuilder.makeLiteral(obj, dataType, obj == null);
+ options.add(literalNode.toProtobuf());
+ }
+ } else {
+ options = new ArrayList<>();
}
+ builder.addAllOptions(options);
Expression.Builder expressionBuilder = Expression.newBuilder();
expressionBuilder.setSingularOrList(builder);
return expressionBuilder.build();
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/expression/PredicateExpressionTransformer.scala b/gluten-substrait/src/main/scala/org/apache/gluten/expression/PredicateExpressionTransformer.scala
index 9f443973a9..3f74cab69a 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/expression/PredicateExpressionTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/expression/PredicateExpressionTransformer.scala
@@ -40,10 +40,18 @@ case class InSetTransformer(
original: InSet)
extends UnaryExpressionTransformer {
override def doTransform(context: SubstraitContext): ExpressionNode = {
- InExpressionTransformer.toTransformer(
- child.doTransform(context),
- original.hset,
- original.child.dataType)
+ val leftNode = child.doTransform(context)
+ val values = original.hset
+ val valueType = original.child.dataType
+
+ // Keep raw literal values and defer building LiteralNode objects until toProtobuf().
+ val rawValues = new java.util.ArrayList[Object](
+ values.toSeq
+ // Sort elements for deterministic behaviours.
+ .sortBy(Literal(_, valueType).toString())
+ .map(_.asInstanceOf[Object])
+ .asJava)
+ ExpressionBuilder.makeSingularOrListNode(leftNode, rawValues, valueType)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 27/38: [GLUTEN][CORE] Share tier-2 test suites under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit a3b3e94d539a1d5ef7879d8dc00c643bb60ba111
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 20:36:37 2026 +0800
[GLUTEN][CORE] Share tier-2 test suites under backends-core
Move 5 backend-symbol-only-different test files (FunctionsValidateSuite,
JsonFunctionsValidateSuite, FuzzerBase, RowToColumnarFuzzer,
ShuffleWriterFuzzer) from backends-velox/src/test to
backends-core/src/test. FuzzerBase now extends the shared
SharedWholeStageTransformerSuite so future backends can reuse it
without subclassing a backend-specific base. The other four were
already backend-symbol-free and migrate as-is.
Generated-by: TraeCli openrouter-3o
---
.../scala/org/apache/gluten/functions/FunctionsValidateSuite.scala | 0
.../org/apache/gluten/functions/JsonFunctionsValidateSuite.scala | 0
.../src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala | 4 ++--
.../src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala | 0
.../src/test/scala/org/apache/gluten/fuzzer/ShuffleWriterFuzzer.scala | 0
5 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/FunctionsValidateSuite.scala b/backends-core/src/test/scala/org/apache/gluten/functions/FunctionsValidateSuite.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/functions/FunctionsValidateSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/functions/FunctionsValidateSuite.scala
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala b/backends-core/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
diff --git a/backends-velox/src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala b/backends-core/src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala
similarity index 96%
rename from backends-velox/src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala
rename to backends-core/src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala
index 95c9e60409..43cae6c0ed 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala
+++ b/backends-core/src/test/scala/org/apache/gluten/fuzzer/FuzzerBase.scala
@@ -17,13 +17,13 @@
package org.apache.gluten.fuzzer
import org.apache.gluten.benchmarks.RandomParquetDataGenerator
-import org.apache.gluten.execution.VeloxWholeStageTransformerSuite
+import org.apache.gluten.execution.SharedWholeStageTransformerSuite
import org.apache.gluten.fuzzer.FuzzerResult.{Failed, OOM, Successful, TestResult}
import org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget
import org.apache.spark.SparkConf
-abstract class FuzzerBase extends VeloxWholeStageTransformerSuite {
+abstract class FuzzerBase extends SharedWholeStageTransformerSuite {
override protected val resourcePath: String = "/tpch-data-parquet"
override protected val fileFormat: String = "parquet"
diff --git a/backends-velox/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala b/backends-core/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala
rename to backends-core/src/test/scala/org/apache/gluten/fuzzer/RowToColumnarFuzzer.scala
diff --git a/backends-velox/src/test/scala/org/apache/gluten/fuzzer/ShuffleWriterFuzzer.scala b/backends-core/src/test/scala/org/apache/gluten/fuzzer/ShuffleWriterFuzzer.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/fuzzer/ShuffleWriterFuzzer.scala
rename to backends-core/src/test/scala/org/apache/gluten/fuzzer/ShuffleWriterFuzzer.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 33/38: [GLUTEN][CORE] Share tier-2 function test suites under backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 26ec3497b84f9a010882f4f9aa4af38d35486670
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 21:56:13 2026 +0800
[GLUTEN][CORE] Share tier-2 function test suites under backends-core
---
backends-core/pom.xml | 16 +++++-
.../gluten/execution/SharedRowToColumnarExec.scala | 14 ++++-
.../metrics/NestedLoopJoinMetricsUpdater.scala | 2 +-
.../gluten/metrics/SharedJoinMetricsUpdater.scala | 60 ++++++++++++++++++++++
.../benchmarks/NativeBenchmarkPlanGenerator.scala | 4 +-
.../functions/ArithmeticAnsiValidateSuite.scala | 0
.../functions/MathFunctionsValidateSuite.scala | 0
.../functions/WindowFunctionsValidateSuite.scala | 0
.../org/apache/gluten/metrics/MetricsUtil.scala | 11 ++++
9 files changed, 100 insertions(+), 7 deletions(-)
diff --git a/backends-core/pom.xml b/backends-core/pom.xml
index 64dcfbe29e..68aa022d77 100644
--- a/backends-core/pom.xml
+++ b/backends-core/pom.xml
@@ -62,6 +62,12 @@
<version>${scala.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>com.google.jimfs</groupId>
+ <artifactId>jimfs</artifactId>
+ <version>1.3.0</version>
+ <scope>compile</scope>
+ </dependency>
<dependency>
<groupId>org.apache.gluten</groupId>
<artifactId>gluten-substrait</artifactId>
@@ -112,7 +118,7 @@
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
<version>${arrow.version}</version>
- <scope>test</scope>
+ <scope>compile</scope>
</dependency>
</dependencies>
@@ -208,7 +214,13 @@
</dependency>
<dependency>
<groupId>org.apache.iceberg</groupId>
- <artifactId>iceberg-spark-runtime-${sparkbundle.version}_${scala.binary.version}</artifactId>
+ <artifactId>iceberg-core</artifactId>
+ <version>${iceberg.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iceberg</groupId>
+ <artifactId>iceberg-api</artifactId>
<version>${iceberg.version}</version>
<scope>provided</scope>
</dependency>
diff --git a/backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala b/backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala
index 5f375f53ae..22fd7a59ef 100644
--- a/backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/execution/SharedRowToColumnarExec.scala
@@ -29,7 +29,8 @@ import org.apache.spark.broadcast.Broadcast
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{UnsafeProjection, UnsafeRow}
-import org.apache.spark.sql.execution.{BroadcastUtils, SparkPlan}
+import org.apache.spark.sql.catalyst.plans.physical.{BroadcastMode, BroadcastPartitioning, Partitioning}
+import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.execution.metric.SQLMetric
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.StructType
@@ -92,7 +93,7 @@ abstract class SharedRowToColumnarExec(child: SparkPlan) extends RowToColumnarEx
val convertTime = longMetric("convertTime")
val numRows = GlutenConfig.get.maxBatchSize
val numBytes = preferredBatchBytes
- val mode = BroadcastUtils.getBroadcastMode(outputPartitioning)
+ val mode = SharedRowToColumnarExec.getBroadcastMode(outputPartitioning)
val relation = child.executeBroadcast()
sparkToBackendUnsafe(
sparkContext,
@@ -114,6 +115,15 @@ abstract class SharedRowToColumnarExec(child: SparkPlan) extends RowToColumnarEx
object SharedRowToColumnarExec {
+ private def getBroadcastMode(partitioning: Partitioning): BroadcastMode = {
+ partitioning match {
+ case BroadcastPartitioning(mode) =>
+ mode
+ case _ =>
+ throw new IllegalArgumentException("Unexpected partitioning: " + partitioning.toString)
+ }
+ }
+
def toColumnarBatchIterator(
it: Iterator[InternalRow],
schema: StructType,
diff --git a/backends-core/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala
index f2a4dfff36..56b57bc61e 100644
--- a/backends-core/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/metrics/NestedLoopJoinMetricsUpdater.scala
@@ -25,7 +25,7 @@ import java.util
import scala.collection.JavaConverters._
class NestedLoopJoinMetricsUpdater(override val metrics: Map[String, SQLMetric])
- extends JoinMetricsUpdaterBase(metrics) {
+ extends SharedJoinMetricsUpdaterBase(metrics) {
val nestedLoopJoinBuildInputRows: SQLMetric = metrics("nestedLoopJoinBuildInputRows")
val nestedLoopJoinBuildOutputRows: SQLMetric = metrics("nestedLoopJoinBuildOutputRows")
diff --git a/backends-core/src/main/scala/org/apache/gluten/metrics/SharedJoinMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/SharedJoinMetricsUpdater.scala
new file mode 100644
index 0000000000..a393ff2562
--- /dev/null
+++ b/backends-core/src/main/scala/org/apache/gluten/metrics/SharedJoinMetricsUpdater.scala
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.metrics
+
+import org.apache.gluten.metrics.Metrics.SingleMetric
+import org.apache.gluten.substrait.JoinParams
+
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import java.util
+
+trait SharedJoinMetricsUpdater extends MetricsUpdater {
+ def updateJoinMetrics(
+ joinMetrics: util.ArrayList[OperatorMetrics],
+ singleMetrics: SingleMetric,
+ joinParams: JoinParams): Unit
+}
+
+abstract class SharedJoinMetricsUpdaterBase(val metrics: Map[String, SQLMetric])
+ extends SharedJoinMetricsUpdater {
+ val postProjectionCpuCount: SQLMetric = metrics("postProjectionCpuCount")
+ val postProjectionWallNanos: SQLMetric = metrics("postProjectionWallNanos")
+ val numOutputRows: SQLMetric = metrics("numOutputRows")
+ val numOutputVectors: SQLMetric = metrics("numOutputVectors")
+ val numOutputBytes: SQLMetric = metrics("numOutputBytes")
+
+ final override def updateJoinMetrics(
+ joinMetrics: util.ArrayList[OperatorMetrics],
+ singleMetrics: SingleMetric,
+ joinParams: JoinParams): Unit = {
+ if (joinParams != null && joinParams.postProjectionNeeded) {
+ val postProjectMetrics = joinMetrics.remove(0)
+ postProjectionCpuCount += postProjectMetrics.cpuCount
+ postProjectionWallNanos += postProjectMetrics.wallNanos
+ numOutputRows += postProjectMetrics.outputRows
+ numOutputVectors += postProjectMetrics.outputVectors
+ numOutputBytes += postProjectMetrics.outputBytes
+ }
+
+ updateJoinMetricsInternal(joinMetrics, joinParams)
+ }
+
+ protected def updateJoinMetricsInternal(
+ joinMetrics: util.ArrayList[OperatorMetrics],
+ joinParams: JoinParams): Unit
+}
diff --git a/backends-velox/src/test/scala/org/apache/gluten/benchmarks/NativeBenchmarkPlanGenerator.scala b/backends-core/src/test/scala/org/apache/gluten/benchmarks/NativeBenchmarkPlanGenerator.scala
similarity index 96%
rename from backends-velox/src/test/scala/org/apache/gluten/benchmarks/NativeBenchmarkPlanGenerator.scala
rename to backends-core/src/test/scala/org/apache/gluten/benchmarks/NativeBenchmarkPlanGenerator.scala
index 6886adbd26..257f24c5ac 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/benchmarks/NativeBenchmarkPlanGenerator.scala
+++ b/backends-core/src/test/scala/org/apache/gluten/benchmarks/NativeBenchmarkPlanGenerator.scala
@@ -17,7 +17,7 @@
package org.apache.gluten.benchmarks
import org.apache.gluten.config.GlutenConfig
-import org.apache.gluten.execution.{VeloxWholeStageTransformerSuite, WholeStageTransformer}
+import org.apache.gluten.execution.{SharedWholeStageTransformerSuite, WholeStageTransformer}
import org.apache.spark.SparkConf
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec
@@ -30,7 +30,7 @@ import java.io.File
object GenerateExample extends Tag("org.apache.gluten.tags.GenerateExample")
-class NativeBenchmarkPlanGenerator extends VeloxWholeStageTransformerSuite {
+class NativeBenchmarkPlanGenerator extends SharedWholeStageTransformerSuite {
override protected val resourcePath: String = "/tpch-data-parquet"
override protected val fileFormat: String = "parquet"
val generatedPlanDir = getClass.getResource("/").getPath + "../../../generated-native-benchmark/"
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala b/backends-core/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/MathFunctionsValidateSuite.scala b/backends-core/src/test/scala/org/apache/gluten/functions/MathFunctionsValidateSuite.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/functions/MathFunctionsValidateSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/functions/MathFunctionsValidateSuite.scala
diff --git a/backends-velox/src/test/scala/org/apache/gluten/functions/WindowFunctionsValidateSuite.scala b/backends-core/src/test/scala/org/apache/gluten/functions/WindowFunctionsValidateSuite.scala
similarity index 100%
rename from backends-velox/src/test/scala/org/apache/gluten/functions/WindowFunctionsValidateSuite.scala
rename to backends-core/src/test/scala/org/apache/gluten/functions/WindowFunctionsValidateSuite.scala
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala b/backends-velox/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala
index 878eb2dcf1..5c04675126 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala
@@ -276,6 +276,17 @@ object MetricsUtil extends Logging {
p
}
ju.updateJoinMetrics(operatorMetrics, metrics.getSingleMetrics, joinParams)
+ case ju: SharedJoinMetricsUpdaterBase =>
+ // JoinRel and CrossRel output two suites of metrics respectively for build and probe.
+ // Therefore, fetch one more suite of metrics here.
+ operatorMetrics.add(metrics.getOperatorMetrics(curMetricsIdx))
+ curMetricsIdx -= 1
+ val joinParams = Option(joinParamsMap.get(operatorIdx)).getOrElse {
+ val p = JoinParams()
+ p.postProjectionNeeded = false
+ p
+ }
+ ju.updateJoinMetrics(operatorMetrics, metrics.getSingleMetrics, joinParams)
case u: UnionMetricsUpdater =>
// Union outputs two suites of metrics respectively.
// Therefore, fetch one more suite of metrics here.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 17/38: [GLUTEN][CORE] Move ColumnarBatchSerializerInstance to backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 3d280271e054e554b9931cd322acfc2bf1256ad2
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Mon Jun 29 21:22:30 2026 +0800
[GLUTEN][CORE] Move ColumnarBatchSerializerInstance to backends-core
Relocate the abstract base class for columnar batch shuffle serialization into backends-core so multiple columnar backends can reuse the same interface definition while keeping concrete serialization implementations backend-local.
---
.../org/apache/gluten/vectorized/ColumnarBatchSerializerInstance.scala | 0
1 file changed, 0 insertions(+), 0 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/vectorized/ColumnarBatchSerializerInstance.scala b/backends-core/src/main/scala/org/apache/gluten/vectorized/ColumnarBatchSerializerInstance.scala
similarity index 100%
rename from backends-velox/src/main/scala/org/apache/gluten/vectorized/ColumnarBatchSerializerInstance.scala
rename to backends-core/src/main/scala/org/apache/gluten/vectorized/ColumnarBatchSerializerInstance.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 08/38: [GLUTEN][CORE] Move overwrite sql-tests to a backend-neutral shared dir
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit b12003c1f1bd9f71631a464397d741bdcdbe3874
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 16:32:52 2026 +0800
[GLUTEN][CORE] Move overwrite sql-tests to a backend-neutral shared dir
The Velox overwrite sql-tests (inputs/results) were stored under a
backend-specific directory (backends-velox/sql-tests). Move them to a
backend-neutral shared directory (sql-tests-gluten) for spark35/40/41,
and provide a default getResourceFilePath in the SQLQueryTestSettings
trait pointing to it, so any backend can reuse the same resources
without duplicating the files. The corresponding VeloxSQLQueryTestSettings
drop their own path override and inherit the shared default; the test
name lists stay backend-specific.
spark33/34 keep their existing shared "sql-tests" directory (shared by
velox/CH and the suite) untouched, and ClickHouse keeps its own
backends-clickhouse/sql-tests. Behavior-preserving for all backends.
Generated-by: TraeCli openrouter-3o
---
.../test/scala/org/apache/gluten/utils/SQLQueryTestSettings.scala | 5 ++++-
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/array.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/bitwise.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/charvarchar.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/count.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/cte.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/date.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-legacy.sql | 0
.../inputs/datetime-parsing-invalid.sql | 0
.../inputs/datetime-parsing-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-parsing.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/hll.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/linear-regression.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/misc-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/percentiles.sql | 0
.../inputs/postgreSQL/window_part1.sql | 0
.../inputs/postgreSQL/window_part3.sql | 0
.../inputs/postgreSQL/window_part4.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/random.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/regexp-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/string-functions.sql | 0
.../inputs/subquery/in-subquery/in-null-semantics.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/table-valued-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_arithmetic.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_cast.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_datetime_functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_element_at.sql | 0
.../inputs/typeCoercion/native/stringCastAndExpressions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/url-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/window.sql | 0
.../sql-tests => sql-tests-gluten}/results/array.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/bitwise.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/charvarchar.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/count.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/datetime-legacy.sql.out | 0
.../results/datetime-parsing-invalid.sql.out | 0
.../results/datetime-parsing-legacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/datetime-parsing.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-ordinal.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/hll.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/interval.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/linear-regression.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/misc-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/percentiles.sql.out | 0
.../results/postgreSQL/window_part1.sql.out | 0
.../results/postgreSQL/window_part3.sql.out | 0
.../results/postgreSQL/window_part4.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/random.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/regexp-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/string-functions.sql.out | 0
.../results/subquery/in-subquery/in-null-semantics.sql.out | 0
.../results/table-valued-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_arithmetic.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_cast.sql.out | 0
.../results/try_datetime_functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_element_at.sql.out | 0
.../results/typeCoercion/native/stringCastAndExpressions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-group-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/url-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/window.sql.out | 0
.../org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | 3 ---
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/array.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/binary.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_base64.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_basic.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_hex.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_hex_discrete.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/bitwise.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/cast.sql | 0
.../inputs/ceil-floor-with-scale-param.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/change-column.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/charvarchar.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/collations.sql | 0
.../inputs/column-resolution-aggregate.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/column-resolution-sort.sql | 0
.../inputs/columnresolution-negative.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/columnresolution-views.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/columnresolution.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/comments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/comparator.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/conditional-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/count.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cross-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/csv-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-command.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-nested.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-nonlegacy.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/cte.sql | 0
.../inputs/current_database_catalog.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/date.sql | 0
.../inputs/datetime-formatting-invalid.sql | 0
.../inputs/datetime-formatting-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-formatting.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-legacy.sql | 0
.../inputs/datetime-parsing-invalid.sql | 0
.../inputs/datetime-parsing-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-parsing.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-special.sql | 0
.../inputs/decimalArithmeticOperations.sql | 0
.../inputs/describe-part-after-analyze.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/describe-query.sql | 0
.../inputs/describe-table-after-alter-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/describe.sql | 0
.../inputs/double-quoted-identifiers-enabled.sql | 0
.../inputs/double-quoted-identifiers.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/except-all.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/except.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/execute-immediate.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/explain-aqe.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/explain-cbo.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/explain.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/extract.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-analytics.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-alias.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-all-duckdb.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-all-mosha.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-filter.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/grouping_set.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/having.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/higher-order-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/hll.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/identifier-clause.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/ignored.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/ilike-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/ilike-any.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/ilike.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/inline-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/inner-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/intersect-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/join-empty-relation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/join-lateral.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/json-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/keywords-enforced.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/keywords.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/like-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/like-any.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/limit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/linear-regression.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/listagg-collations.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/listagg.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/literals.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/map.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/mask-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/math.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/misc-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/mode.sql | 0
.../inputs/named-function-arguments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/natural-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/non-excludable-rule.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/array.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/cast.sql | 0
.../inputs/nonansi/conditional-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/date.sql | 0
.../inputs/nonansi/datetime-parsing-invalid.sql | 0
.../inputs/nonansi/datetime-special.sql | 0
.../inputs/nonansi/decimalArithmeticOperations.sql | 0
.../inputs/nonansi/double-quoted-identifiers.sql | 0
.../inputs/nonansi/higher-order-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/keywords.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/literals.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/map.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/math.sql | 0
.../inputs/nonansi/parse-schema-string.sql | 0
.../inputs/nonansi/string-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/try_aggregates.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/try_arithmetic.sql | 0
.../inputs/nonansi/try_datetime_functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/try_element_at.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/null-handling.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/null-propagation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/operators.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by-all.sql | 0
.../inputs/order-by-nulls-ordering.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/outer-join.sql | 0
.../inputs/parse-query-correctness-old-behavior.sql | 0
.../inputs/parse-query-correctness.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/parse-schema-string.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/percentiles.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/pipe-operators.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/pivot.sql | 0
.../inputs/postgreSQL/aggregates_part1.sql | 0
.../inputs/postgreSQL/aggregates_part2.sql | 0
.../inputs/postgreSQL/aggregates_part3.sql | 0
.../inputs/postgreSQL/aggregates_part4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/boolean.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/case.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/comments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/create_view.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/date.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/float4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/float8.sql | 0
.../inputs/postgreSQL/groupingsets.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/insert.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/int2.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/int4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/int8.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/limit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/numeric.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/select.sql | 0
.../inputs/postgreSQL/select_distinct.sql | 0
.../inputs/postgreSQL/select_having.sql | 0
.../inputs/postgreSQL/select_implicit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/strings.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/text.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/union.sql | 0
.../inputs/postgreSQL/window_part1.sql | 0
.../inputs/postgreSQL/window_part2.sql | 0
.../inputs/postgreSQL/window_part3.sql | 0
.../inputs/postgreSQL/window_part4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/with.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/pred-pushdown.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/predicate-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/query_regex_column.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/random.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/regexp-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/selectExcept.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-create-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-tables.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-tblproperties.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-views.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show_columns.sql | 0
.../inputs/sql-compatibility-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/sql-on-files.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/sql-session-variables.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/sql-udf.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/string-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/struct.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/subexp-elimination.sql | 0
.../inputs/subquery/exists-subquery/exists-aggregate.sql | 0
.../inputs/subquery/exists-subquery/exists-basic.sql | 0
.../inputs/subquery/exists-subquery/exists-count-bug.sql | 0
.../inputs/subquery/exists-subquery/exists-cte.sql | 0
.../inputs/subquery/exists-subquery/exists-having.sql | 0
.../inputs/subquery/exists-subquery/exists-in-join-condition.sql | 0
.../inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql | 0
.../inputs/subquery/exists-subquery/exists-orderby-limit.sql | 0
.../inputs/subquery/exists-subquery/exists-outside-filter.sql | 0
.../inputs/subquery/exists-subquery/exists-within-and-or.sql | 0
.../inputs/subquery/in-subquery/in-basic.sql | 0
.../inputs/subquery/in-subquery/in-count-bug.sql | 0
.../inputs/subquery/in-subquery/in-group-by.sql | 0
.../inputs/subquery/in-subquery/in-having.sql | 0
.../inputs/subquery/in-subquery/in-joins.sql | 0
.../inputs/subquery/in-subquery/in-limit.sql | 0
.../inputs/subquery/in-subquery/in-multiple-columns.sql | 0
.../inputs/subquery/in-subquery/in-null-semantics.sql | 0
.../inputs/subquery/in-subquery/in-nullability.sql | 0
.../inputs/subquery/in-subquery/in-order-by.sql | 0
.../inputs/subquery/in-subquery/in-set-operations.sql | 0
.../inputs/subquery/in-subquery/in-subquery-in-join-condition.sql | 0
.../inputs/subquery/in-subquery/in-with-cte.sql | 0
.../inputs/subquery/in-subquery/nested-not-in.sql | 0
.../inputs/subquery/in-subquery/not-in-group-by.sql | 0
.../inputs/subquery/in-subquery/not-in-joins.sql | 0
.../subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql | 0
.../inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql | 0
.../subquery/in-subquery/not-in-unit-tests-single-column-literal.sql | 0
.../inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql | 0
.../inputs/subquery/in-subquery/simple-in.sql | 0
.../inputs/subquery/negative-cases/invalid-correlation.sql | 0
.../inputs/subquery/negative-cases/subq-input-typecheck.sql | 0
.../subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-select.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql | 0
.../inputs/subquery/subquery-in-from.sql | 0
.../inputs/subquery/subquery-nested-data.sql | 0
.../inputs/subquery/subquery-offset.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/table-aliases.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/table-valued-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/tablesample-negative.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestamp-ltz.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestamp.sql | 0
.../inputs/timestampNTZ/datetime-special-ansi.sql | 0
.../inputs/timestampNTZ/datetime-special.sql | 0
.../inputs/timestampNTZ/timestamp-ansi.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestampNTZ/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timezone.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/to_from_avro.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/transform.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try-string-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_aggregates.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_arithmetic.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_cast.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_datetime_functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_element_at.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_reflect.sql | 0
.../inputs/typeCoercion/native/arrayJoin.sql | 0
.../inputs/typeCoercion/native/binaryComparison.sql | 0
.../inputs/typeCoercion/native/booleanEquality.sql | 0
.../inputs/typeCoercion/native/caseWhenCoercion.sql | 0
.../inputs/typeCoercion/native/concat.sql | 0
.../inputs/typeCoercion/native/dateTimeOperations.sql | 0
.../inputs/typeCoercion/native/decimalPrecision.sql | 0
.../inputs/typeCoercion/native/division.sql | 0
.../inputs/typeCoercion/native/elt.sql | 0
.../inputs/typeCoercion/native/ifCoercion.sql | 0
.../inputs/typeCoercion/native/implicitTypeCasts.sql | 0
.../inputs/typeCoercion/native/inConversion.sql | 0
.../inputs/typeCoercion/native/mapZipWith.sql | 0
.../inputs/typeCoercion/native/mapconcat.sql | 0
.../inputs/typeCoercion/native/promoteStrings.sql | 0
.../inputs/typeCoercion/native/stringCastAndExpressions.sql | 0
.../inputs/typeCoercion/native/widenSetOperationTypes.sql | 0
.../inputs/typeCoercion/native/windowFrameCoercion.sql | 0
.../inputs/udaf/udaf-group-analytics.sql | 0
.../inputs/udaf/udaf-group-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udaf/udaf-group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udaf/udaf-grouping-set.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udaf/udaf.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part1.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part2.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part3.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part4.sql | 0
.../inputs/udf/postgreSQL/udf-case.sql | 0
.../inputs/udf/postgreSQL/udf-join.sql | 0
.../inputs/udf/postgreSQL/udf-select_having.sql | 0
.../inputs/udf/postgreSQL/udf-select_implicit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-count.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-cross-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-except-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-except.sql | 0
.../inputs/udf/udf-group-analytics.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-having.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-inline-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-inner-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-intersect-all.sql | 0
.../inputs/udf/udf-join-empty-relation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-natural-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-outer-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-pivot.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-special-values.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-udaf.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-union.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-window.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udtf/udtf.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/union.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/unpivot.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/url-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/using-join.sql | 0
.../inputs/variant/named-function-arguments.sql | 0
.../inputs/view-schema-binding-config.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/view-schema-binding.sql | 0
.../inputs/view-schema-compensation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/view-schema-evolution.sql | 0
.../inputs/view-schema-type-evolution.sql | 0
.../inputs/view-with-default-collation.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/window.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/xml-functions.sql | 0
.../sql-tests => sql-tests-gluten}/results/array.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary_base64.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary_basic.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary_hex.sql.out | 0
.../results/binary_hex_discrete.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/bitwise.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cast.sql.out | 0
.../results/ceil-floor-with-scale-param.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/change-column.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/charvarchar.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/collations.sql.out | 0
.../results/column-resolution-aggregate.sql.out | 0
.../results/column-resolution-sort.sql.out | 0
.../results/columnresolution-negative.sql.out | 0
.../results/columnresolution-views.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/columnresolution.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/comments.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/comparator.sql.out | 0
.../results/conditional-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/count.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cross-join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/csv-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-command.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-legacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-nested.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-nonlegacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte.sql.out | 0
.../results/current_database_catalog.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/date.sql.out | 0
.../results/datetime-formatting-invalid.sql.out | 0
.../results/datetime-formatting-legacy.sql.out | 0
.../results/datetime-formatting.sql.out | 0
.../results/datetime-formatting.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/datetime-legacy.sql.out | 0
.../results/datetime-parsing-invalid.sql.out | 0
.../results/datetime-parsing-legacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/datetime-parsing.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/datetime-special.sql.out | 0
.../results/decimalArithmeticOperations.sql.out | 0
.../results/describe-part-after-analyze.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/describe-query.sql.out | 0
.../results/describe-table-after-alter-table.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/describe.sql.out | 0
.../results/double-quoted-identifiers-enabled.sql.out | 0
.../results/double-quoted-identifiers.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/except-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/except.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/execute-immediate.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/explain-aqe.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/explain-cbo.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/explain.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/extract.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-analytics.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-alias.sql.out | 0
.../results/group-by-all-duckdb.sql.out | 0
.../results/group-by-all-mosha.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-filter.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-ordinal.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/grouping_set.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/having.sql.out | 0
.../results/higher-order-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/hll.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/identifier-clause.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/ilike-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/ilike-any.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/ilike.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/inline-table.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/inner-join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/intersect-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/interval.sql.out | 0
.../results/join-empty-relation.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/join-lateral.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/json-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/keywords-enforced.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/keywords.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/like-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/like-any.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/limit.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/linear-regression.sql.out | 0
.../results/listagg-collations.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/listagg.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/literals.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/map.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/mask-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/math.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/misc-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/mode.sql.out | 0
.../results/named-function-arguments.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/natural-join.sql.out | 0
.../results/non-excludable-rule.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/array.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/cast.sql.out | 0
.../results/nonansi/conditional-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/date.sql.out | 0
.../results/nonansi/datetime-parsing-invalid.sql.out | 0
.../results/nonansi/datetime-special.sql.out | 0
.../results/nonansi/decimalArithmeticOperations.sql.out | 0
.../results/nonansi/double-quoted-identifiers.sql.out | 0
.../results/nonansi/higher-order-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/interval.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/keywords.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/literals.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/map.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/math.sql.out | 0
.../results/nonansi/parse-schema-string.sql.out | 0
.../results/nonansi/string-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/timestamp.sql.out | 0
.../results/nonansi/try_aggregates.sql.out | 0
.../results/nonansi/try_aggregates.sql.out.java21 | 0
.../results/nonansi/try_arithmetic.sql.out | 0
.../results/nonansi/try_arithmetic.sql.out.java21 | 0
.../results/nonansi/try_datetime_functions.sql.out | 0
.../results/nonansi/try_element_at.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/null-handling.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/null-propagation.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/operators.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by-all.sql.out | 0
.../results/order-by-nulls-ordering.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by-ordinal.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/outer-join.sql.out | 0
.../results/parse-query-correctness-old-behavior.sql.out | 0
.../results/parse-query-correctness.sql.out | 0
.../results/parse-schema-string.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/percentiles.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/pipe-operators.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/pivot.sql.out | 0
.../results/postgreSQL/aggregates_part1.sql.out | 0
.../results/postgreSQL/aggregates_part2.sql.out | 0
.../results/postgreSQL/aggregates_part3.sql.out | 0
.../results/postgreSQL/aggregates_part4.sql.out | 0
.../results/postgreSQL/boolean.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/case.sql.out | 0
.../results/postgreSQL/comments.sql.out | 0
.../results/postgreSQL/create_view.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/date.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/float4.sql.out | 0
.../results/postgreSQL/float4.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/float8.sql.out | 0
.../results/postgreSQL/groupingsets.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/insert.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/int2.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/int4.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/int8.sql.out | 0
.../results/postgreSQL/int8.sql.out.java21 | 0
.../results/postgreSQL/interval.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/limit.sql.out | 0
.../results/postgreSQL/numeric.sql.out | 0
.../results/postgreSQL/numeric.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/select.sql.out | 0
.../results/postgreSQL/select_distinct.sql.out | 0
.../results/postgreSQL/select_having.sql.out | 0
.../results/postgreSQL/select_implicit.sql.out | 0
.../results/postgreSQL/strings.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/text.sql.out | 0
.../results/postgreSQL/timestamp.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/union.sql.out | 0
.../results/postgreSQL/window_part1.sql.out | 0
.../results/postgreSQL/window_part2.sql.out | 0
.../results/postgreSQL/window_part3.sql.out | 0
.../results/postgreSQL/window_part4.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/with.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/pred-pushdown.sql.out | 0
.../results/predicate-functions.sql.out | 0
.../results/query_regex_column.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/random.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/regexp-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/selectExcept.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show-create-table.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show-tables.sql.out | 0
.../results/show-tblproperties.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show-views.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show_columns.sql.out | 0
.../results/sql-compatibility-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/sql-on-files.sql.out | 0
.../results/sql-session-variables.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/sql-udf.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/string-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/struct.sql.out | 0
.../results/subexp-elimination.sql.out | 0
.../results/subquery/exists-subquery/exists-aggregate.sql.out | 0
.../results/subquery/exists-subquery/exists-basic.sql.out | 0
.../results/subquery/exists-subquery/exists-count-bug.sql.out | 0
.../results/subquery/exists-subquery/exists-cte.sql.out | 0
.../results/subquery/exists-subquery/exists-having.sql.out | 0
.../subquery/exists-subquery/exists-in-join-condition.sql.out | 0
.../subquery/exists-subquery/exists-joins-and-set-ops.sql.out | 0
.../results/subquery/exists-subquery/exists-orderby-limit.sql.out | 0
.../results/subquery/exists-subquery/exists-outside-filter.sql.out | 0
.../results/subquery/exists-subquery/exists-within-and-or.sql.out | 0
.../results/subquery/in-subquery/in-basic.sql.out | 0
.../results/subquery/in-subquery/in-count-bug.sql.out | 0
.../results/subquery/in-subquery/in-group-by.sql.out | 0
.../results/subquery/in-subquery/in-having.sql.out | 0
.../results/subquery/in-subquery/in-joins.sql.out | 0
.../results/subquery/in-subquery/in-limit.sql.out | 0
.../results/subquery/in-subquery/in-multiple-columns.sql.out | 0
.../results/subquery/in-subquery/in-null-semantics.sql.out | 0
.../results/subquery/in-subquery/in-nullability.sql.out | 0
.../results/subquery/in-subquery/in-order-by.sql.out | 0
.../results/subquery/in-subquery/in-set-operations.sql.out | 0
.../subquery/in-subquery/in-subquery-in-join-condition.sql.out | 0
.../results/subquery/in-subquery/in-with-cte.sql.out | 0
.../results/subquery/in-subquery/nested-not-in.sql.out | 0
.../results/subquery/in-subquery/not-in-group-by.sql.out | 0
.../results/subquery/in-subquery/not-in-joins.sql.out | 0
.../in-subquery/not-in-unit-tests-multi-column-literal.sql.out | 0
.../subquery/in-subquery/not-in-unit-tests-multi-column.sql.out | 0
.../in-subquery/not-in-unit-tests-single-column-literal.sql.out | 0
.../subquery/in-subquery/not-in-unit-tests-single-column.sql.out | 0
.../results/subquery/in-subquery/simple-in.sql.out | 0
.../results/subquery/negative-cases/invalid-correlation.sql.out | 0
.../results/subquery/negative-cases/subq-input-typecheck.sql.out | 0
.../scalar-subquery/nested-scalar-subquery-count-bug.sql.out | 0
.../subquery/scalar-subquery/scalar-subquery-count-bug.sql.out | 0
.../subquery/scalar-subquery/scalar-subquery-group-by.sql.out | 0
.../subquery/scalar-subquery/scalar-subquery-predicate.sql.out | 0
.../results/subquery/scalar-subquery/scalar-subquery-select.sql.out | 0
.../results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out | 0
.../results/subquery/subquery-in-from.sql.out | 0
.../results/subquery/subquery-nested-data.sql.out | 0
.../results/subquery/subquery-offset.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/table-aliases.sql.out | 0
.../results/table-valued-functions.sql.out | 0
.../results/tablesample-negative.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timestamp-ltz.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timestamp-ntz.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timestamp.sql.out | 0
.../results/timestampNTZ/datetime-special-ansi.sql.out | 0
.../results/timestampNTZ/datetime-special.sql.out | 0
.../results/timestampNTZ/timestamp-ansi.sql.out | 0
.../results/timestampNTZ/timestamp.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timezone.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/to_from_avro.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/transform.sql.out | 0
.../results/try-string-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_aggregates.sql.out | 0
.../results/try_aggregates.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/try_arithmetic.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_cast.sql.out | 0
.../results/try_datetime_functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_element_at.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_reflect.sql.out | 0
.../results/typeCoercion/native/arrayJoin.sql.out | 0
.../results/typeCoercion/native/binaryComparison.sql.out | 0
.../results/typeCoercion/native/booleanEquality.sql.out | 0
.../results/typeCoercion/native/caseWhenCoercion.sql.out | 0
.../results/typeCoercion/native/concat.sql.out | 0
.../results/typeCoercion/native/dateTimeOperations.sql.out | 0
.../results/typeCoercion/native/decimalPrecision.sql.out | 0
.../results/typeCoercion/native/division.sql.out | 0
.../results/typeCoercion/native/elt.sql.out | 0
.../results/typeCoercion/native/ifCoercion.sql.out | 0
.../results/typeCoercion/native/implicitTypeCasts.sql.out | 0
.../results/typeCoercion/native/inConversion.sql.out | 0
.../results/typeCoercion/native/mapZipWith.sql.out | 0
.../results/typeCoercion/native/mapconcat.sql.out | 0
.../results/typeCoercion/native/promoteStrings.sql.out | 0
.../results/typeCoercion/native/stringCastAndExpressions.sql.out | 0
.../results/typeCoercion/native/widenSetOperationTypes.sql.out | 0
.../results/typeCoercion/native/windowFrameCoercion.sql.out | 0
.../results/udaf/udaf-group-analytics.sql.out | 0
.../results/udaf/udaf-group-by-ordinal.sql.out | 0
.../results/udaf/udaf-group-by.sql.out | 0
.../results/udaf/udaf-grouping-set.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udaf/udaf.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part1.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part2.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part3.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part4.sql.out | 0
.../results/udf/postgreSQL/udf-case.sql.out | 0
.../results/udf/postgreSQL/udf-join.sql.out | 0
.../results/udf/postgreSQL/udf-select_having.sql.out | 0
.../results/udf/postgreSQL/udf-select_implicit.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-count.sql.out | 0
.../results/udf/udf-cross-join.sql.out | 0
.../results/udf/udf-except-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-except.sql.out | 0
.../results/udf/udf-group-analytics.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-group-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-having.sql.out | 0
.../results/udf/udf-inline-table.sql.out | 0
.../results/udf/udf-inner-join.sql.out | 0
.../results/udf/udf-intersect-all.sql.out | 0
.../results/udf/udf-join-empty-relation.sql.out | 0
.../results/udf/udf-natural-join.sql.out | 0
.../results/udf/udf-outer-join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-pivot.sql.out | 0
.../results/udf/udf-special-values.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-udaf.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-union.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-window.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udtf/udtf.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/union.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/unpivot.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/url-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/using-join.sql.out | 0
.../results/variant/named-function-arguments.sql.out | 0
.../results/view-schema-binding-config.sql.out | 0
.../results/view-schema-binding.sql.out | 0
.../results/view-schema-compensation.sql.out | 0
.../results/view-schema-evolution.sql.out | 0
.../results/view-schema-type-evolution.sql.out | 0
.../results/view-with-default-collation.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/window.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/xml-functions.sql.out | 0
.../org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | 3 ---
.../inputs/alias-resolution-edge-cases.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/array.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/binary.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_base64.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_basic.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_hex.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/binary_hex_discrete.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/bitwise.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/cast.sql | 0
.../inputs/ceil-floor-with-scale-param.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/change-column.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/charvarchar.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/collations-aliases.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/collations-basic.sql | 0
.../inputs/collations-padding-trim.sql | 0
.../inputs/collations-string-functions.sql | 0
.../inputs/column-resolution-aggregate.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/column-resolution-sort.sql | 0
.../inputs/columnresolution-negative.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/columnresolution-views.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/columnresolution.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/comments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/comparator.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/conditional-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/count.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cross-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/csv-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-command.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-nested.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-nonlegacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/cte-recursion.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/cte.sql | 0
.../inputs/current_database_catalog.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/date.sql | 0
.../inputs/datetime-formatting-invalid.sql | 0
.../inputs/datetime-formatting-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-formatting.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-legacy.sql | 0
.../inputs/datetime-parsing-invalid.sql | 0
.../inputs/datetime-parsing-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-parsing.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/datetime-special.sql | 0
.../inputs/decimalArithmeticOperations.sql | 0
.../inputs/describe-part-after-analyze.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/describe-query.sql | 0
.../inputs/describe-table-after-alter-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/describe.sql | 0
.../inputs/double-quoted-identifiers-enabled.sql | 0
.../inputs/double-quoted-identifiers.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/except-all.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/except.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/execute-immediate.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/explain-aqe.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/explain-cbo.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/explain.sql | 0
.../inputs/extract-value-resolution-edge-cases.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/extract.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/generators.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-analytics.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-alias.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-all-duckdb.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-all-mosha.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-filter.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/grouping_set.sql | 0
.../inputs/having-and-order-by-recursive-type-name-resolution.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/having.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/higher-order-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/hll.sql | 0
.../inputs/identifier-clause-legacy.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/identifier-clause.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/ignored.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/ilike-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/ilike-any.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/ilike.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/inline-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/inner-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/intersect-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/join-empty-relation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/join-lateral.sql | 0
.../inputs/join-resolution-edge-cases.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/json-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/keywords-enforced.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/keywords.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/kllquantiles.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/like-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/like-any.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/limit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/linear-regression.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/listagg-collations.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/listagg.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/literals.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/map.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/mask-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/math.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/misc-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/mode.sql | 0
...precedence-in-order-by-and-having-with-conflicting-attributes.sql | 0
.../inputs/named-function-arguments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/natural-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/non-excludable-rule.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/array.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/cast.sql | 0
.../inputs/nonansi/conditional-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/date.sql | 0
.../inputs/nonansi/datetime-parsing-invalid.sql | 0
.../inputs/nonansi/datetime-special.sql | 0
.../inputs/nonansi/decimalArithmeticOperations.sql | 0
.../inputs/nonansi/double-quoted-identifiers.sql | 0
.../inputs/nonansi/higher-order-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/keywords.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/literals.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/map.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/math.sql | 0
.../inputs/nonansi/parse-schema-string.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/st-functions.sql | 0
.../inputs/nonansi/string-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/try_aggregates.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/try_arithmetic.sql | 0
.../inputs/nonansi/try_datetime_functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/nonansi/try_element_at.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/null-handling.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/null-propagation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/operators.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by-alias.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by-all.sql | 0
.../inputs/order-by-and-having-on-top-of-aggregate-with-join.sql | 0
.../inputs/order-by-nulls-ordering.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/order-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/outer-join.sql | 0
.../inputs/parse-query-correctness-old-behavior.sql | 0
.../inputs/parse-query-correctness.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/parse-schema-string.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/percentiles.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/pipe-operators.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/pivot.sql | 0
.../inputs/postgreSQL/aggregates_part1.sql | 0
.../inputs/postgreSQL/aggregates_part2.sql | 0
.../inputs/postgreSQL/aggregates_part3.sql | 0
.../inputs/postgreSQL/aggregates_part4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/boolean.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/case.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/comments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/create_view.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/date.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/float4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/float8.sql | 0
.../inputs/postgreSQL/groupingsets.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/insert.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/int2.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/int4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/int8.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/interval.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/limit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/numeric.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/select.sql | 0
.../inputs/postgreSQL/select_distinct.sql | 0
.../inputs/postgreSQL/select_having.sql | 0
.../inputs/postgreSQL/select_implicit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/strings.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/text.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/union.sql | 0
.../inputs/postgreSQL/window_part1.sql | 0
.../inputs/postgreSQL/window_part2.sql | 0
.../inputs/postgreSQL/window_part3.sql | 0
.../inputs/postgreSQL/window_part4.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/postgreSQL/with.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/pred-pushdown.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/predicate-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/query_regex_column.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/random.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/regexp-functions.sql | 0
.../inputs/replacing-missing-expression-with-alias.sql | 0
.../inputs/runtime-replaceable-edge-cases.sql | 0
.../inputs/scripting/randomly_generated_scripts.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/selectExcept.sql | 0
.../inputs/session-variable-precedence.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-create-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-tables.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-tblproperties.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show-views.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/show_columns.sql | 0
.../inputs/sql-compatibility-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/sql-on-files.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/sql-session-variables.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/sql-udf.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/st-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/string-functions.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/struct.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/subexp-elimination.sql | 0
.../inputs/subquery/exists-subquery/exists-aggregate.sql | 0
.../inputs/subquery/exists-subquery/exists-basic.sql | 0
.../inputs/subquery/exists-subquery/exists-count-bug.sql | 0
.../inputs/subquery/exists-subquery/exists-cte.sql | 0
.../inputs/subquery/exists-subquery/exists-having.sql | 0
.../inputs/subquery/exists-subquery/exists-in-join-condition.sql | 0
.../inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql | 0
.../inputs/subquery/exists-subquery/exists-orderby-limit.sql | 0
.../inputs/subquery/exists-subquery/exists-outside-filter.sql | 0
.../inputs/subquery/exists-subquery/exists-within-and-or.sql | 0
.../inputs/subquery/in-subquery/in-basic.sql | 0
.../inputs/subquery/in-subquery/in-count-bug.sql | 0
.../inputs/subquery/in-subquery/in-group-by.sql | 0
.../inputs/subquery/in-subquery/in-having.sql | 0
.../inputs/subquery/in-subquery/in-joins.sql | 0
.../inputs/subquery/in-subquery/in-limit.sql | 0
.../inputs/subquery/in-subquery/in-multiple-columns.sql | 0
.../inputs/subquery/in-subquery/in-null-semantics.sql | 0
.../inputs/subquery/in-subquery/in-nullability.sql | 0
.../inputs/subquery/in-subquery/in-order-by.sql | 0
.../inputs/subquery/in-subquery/in-set-operations.sql | 0
.../inputs/subquery/in-subquery/in-subquery-in-join-condition.sql | 0
.../inputs/subquery/in-subquery/in-with-cte.sql | 0
.../inputs/subquery/in-subquery/nested-not-in.sql | 0
.../inputs/subquery/in-subquery/not-in-group-by.sql | 0
.../inputs/subquery/in-subquery/not-in-joins.sql | 0
.../subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql | 0
.../inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql | 0
.../subquery/in-subquery/not-in-unit-tests-single-column-literal.sql | 0
.../inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql | 0
.../inputs/subquery/in-subquery/simple-in.sql | 0
.../inputs/subquery/negative-cases/invalid-correlation.sql | 0
.../inputs/subquery/negative-cases/subq-input-typecheck.sql | 0
.../inputs/subquery/resolution-edge-cases.sql | 0
.../subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-select.sql | 0
.../inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql | 0
.../inputs/subquery/subquery-in-from.sql | 0
.../inputs/subquery/subquery-nested-data.sql | 0
.../inputs/subquery/subquery-offset.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/table-aliases.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/table-valued-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/tablesample-negative.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/thetasketch.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/time.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestamp-ltz.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestamp.sql | 0
.../inputs/timestampNTZ/datetime-special-ansi.sql | 0
.../inputs/timestampNTZ/datetime-special.sql | 0
.../inputs/timestampNTZ/timestamp-ansi.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timestampNTZ/timestamp.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/timezone.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/to_from_avro.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/transform.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try-string-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_aggregates.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_arithmetic.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_cast.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_datetime_functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_element_at.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/try_reflect.sql | 0
.../inputs/type-coercion-edge-cases.sql | 0
.../inputs/typeCoercion/native/arrayJoin.sql | 0
.../inputs/typeCoercion/native/binaryComparison.sql | 0
.../inputs/typeCoercion/native/booleanEquality.sql | 0
.../inputs/typeCoercion/native/caseWhenCoercion.sql | 0
.../inputs/typeCoercion/native/concat.sql | 0
.../inputs/typeCoercion/native/dateTimeOperations.sql | 0
.../inputs/typeCoercion/native/decimalPrecision.sql | 0
.../inputs/typeCoercion/native/division.sql | 0
.../inputs/typeCoercion/native/elt.sql | 0
.../inputs/typeCoercion/native/ifCoercion.sql | 0
.../inputs/typeCoercion/native/implicitTypeCasts.sql | 0
.../inputs/typeCoercion/native/inConversion.sql | 0
.../inputs/typeCoercion/native/mapZipWith.sql | 0
.../inputs/typeCoercion/native/mapconcat.sql | 0
.../inputs/typeCoercion/native/promoteStrings.sql | 0
.../inputs/typeCoercion/native/stringCastAndExpressions.sql | 0
.../inputs/typeCoercion/native/widenSetOperationTypes.sql | 0
.../inputs/typeCoercion/native/windowFrameCoercion.sql | 0
.../inputs/udaf/udaf-group-analytics.sql | 0
.../inputs/udaf/udaf-group-by-ordinal.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udaf/udaf-group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udaf/udaf-grouping-set.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udaf/udaf.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part1.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part2.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part3.sql | 0
.../inputs/udf/postgreSQL/udf-aggregates_part4.sql | 0
.../inputs/udf/postgreSQL/udf-case.sql | 0
.../inputs/udf/postgreSQL/udf-join.sql | 0
.../inputs/udf/postgreSQL/udf-select_having.sql | 0
.../inputs/udf/postgreSQL/udf-select_implicit.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-count.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-cross-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-except-all.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-except.sql | 0
.../inputs/udf/udf-group-analytics.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-group-by.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-having.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-inline-table.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-inner-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-intersect-all.sql | 0
.../inputs/udf/udf-join-empty-relation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-natural-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-outer-join.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-pivot.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-special-values.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-udaf.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-union.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udf/udf-window.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/udtf/udtf.sql | 0
.../inputs/union-per-child-output-deduplication.sql | 0
.../inputs/union-resolution-edge-cases.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/union.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/unpivot.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/url-functions.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/using-join.sql | 0
.../inputs/variant-field-extractions.sql | 0
.../inputs/variant/named-function-arguments.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/view-conf.sql | 0
.../inputs/view-schema-binding-config.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/view-schema-binding.sql | 0
.../inputs/view-schema-compensation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/view-schema-evolution.sql | 0
.../inputs/view-schema-type-evolution.sql | 0
.../inputs/view-with-default-collation.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/window-with-aggregates.sql | 0
.../{backends-velox/sql-tests => sql-tests-gluten}/inputs/window.sql | 0
.../sql-tests => sql-tests-gluten}/inputs/xml-functions.sql | 0
.../results/alias-resolution-edge-cases.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/array.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary_base64.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary_basic.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/binary_hex.sql.out | 0
.../results/binary_hex_discrete.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/bitwise.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cast.sql.out | 0
.../results/ceil-floor-with-scale-param.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/change-column.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/charvarchar.sql.out | 0
.../results/collations-aliases.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/collations-basic.sql.out | 0
.../results/collations-padding-trim.sql.out | 0
.../results/collations-string-functions.sql.out | 0
.../results/column-resolution-aggregate.sql.out | 0
.../results/column-resolution-sort.sql.out | 0
.../results/columnresolution-negative.sql.out | 0
.../results/columnresolution-views.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/columnresolution.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/comments.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/comparator.sql.out | 0
.../results/conditional-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/count.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cross-join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/csv-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-command.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-legacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-nested.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-nonlegacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte-recursion.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/cte.sql.out | 0
.../results/current_database_catalog.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/date.sql.out | 0
.../results/datetime-formatting-invalid.sql.out | 0
.../results/datetime-formatting-legacy.sql.out | 0
.../results/datetime-formatting.sql.out | 0
.../results/datetime-formatting.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/datetime-legacy.sql.out | 0
.../results/datetime-parsing-invalid.sql.out | 0
.../results/datetime-parsing-legacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/datetime-parsing.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/datetime-special.sql.out | 0
.../results/decimalArithmeticOperations.sql.out | 0
.../results/describe-part-after-analyze.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/describe-query.sql.out | 0
.../results/describe-table-after-alter-table.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/describe.sql.out | 0
.../results/double-quoted-identifiers-enabled.sql.out | 0
.../results/double-quoted-identifiers.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/except-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/except.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/execute-immediate.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/explain-aqe.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/explain-cbo.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/explain.sql.out | 0
.../results/extract-value-resolution-edge-cases.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/extract.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/generators.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-analytics.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-alias.sql.out | 0
.../results/group-by-all-duckdb.sql.out | 0
.../results/group-by-all-mosha.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-filter.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by-ordinal.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/group-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/grouping_set.sql.out | 0
.../having-and-order-by-recursive-type-name-resolution.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/having.sql.out | 0
.../results/higher-order-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/hll.sql.out | 0
.../results/identifier-clause-legacy.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/identifier-clause.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/ilike-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/ilike-any.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/ilike.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/inline-table.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/inner-join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/intersect-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/interval.sql.out | 0
.../results/join-empty-relation.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/join-lateral.sql.out | 0
.../results/join-resolution-edge-cases.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/json-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/keywords-enforced.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/keywords.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/kllquantiles.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/like-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/like-any.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/limit.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/linear-regression.sql.out | 0
.../results/listagg-collations.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/listagg.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/literals.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/map.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/mask-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/math.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/misc-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/mode.sql.out | 0
...edence-in-order-by-and-having-with-conflicting-attributes.sql.out | 0
.../results/named-function-arguments.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/natural-join.sql.out | 0
.../results/non-excludable-rule.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/array.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/cast.sql.out | 0
.../results/nonansi/conditional-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/date.sql.out | 0
.../results/nonansi/datetime-parsing-invalid.sql.out | 0
.../results/nonansi/datetime-special.sql.out | 0
.../results/nonansi/decimalArithmeticOperations.sql.out | 0
.../results/nonansi/double-quoted-identifiers.sql.out | 0
.../results/nonansi/higher-order-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/interval.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/keywords.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/literals.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/map.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/math.sql.out | 0
.../results/nonansi/parse-schema-string.sql.out | 0
.../results/nonansi/st-functions.sql.out | 0
.../results/nonansi/string-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/nonansi/timestamp.sql.out | 0
.../results/nonansi/try_aggregates.sql.out | 0
.../results/nonansi/try_aggregates.sql.out.java21 | 0
.../results/nonansi/try_arithmetic.sql.out | 0
.../results/nonansi/try_arithmetic.sql.out.java21 | 0
.../results/nonansi/try_datetime_functions.sql.out | 0
.../results/nonansi/try_element_at.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/null-handling.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/null-propagation.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/operators.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by-alias.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by-all.sql.out | 0
.../order-by-and-having-on-top-of-aggregate-with-join.sql.out | 0
.../results/order-by-nulls-ordering.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by-ordinal.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/order-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/outer-join.sql.out | 0
.../results/parse-query-correctness-old-behavior.sql.out | 0
.../results/parse-query-correctness.sql.out | 0
.../results/parse-schema-string.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/percentiles.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/pipe-operators.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/pivot.sql.out | 0
.../results/postgreSQL/aggregates_part1.sql.out | 0
.../results/postgreSQL/aggregates_part2.sql.out | 0
.../results/postgreSQL/aggregates_part3.sql.out | 0
.../results/postgreSQL/aggregates_part4.sql.out | 0
.../results/postgreSQL/boolean.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/case.sql.out | 0
.../results/postgreSQL/comments.sql.out | 0
.../results/postgreSQL/create_view.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/date.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/float4.sql.out | 0
.../results/postgreSQL/float4.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/float8.sql.out | 0
.../results/postgreSQL/groupingsets.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/insert.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/int2.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/int4.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/int8.sql.out | 0
.../results/postgreSQL/int8.sql.out.java21 | 0
.../results/postgreSQL/interval.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/limit.sql.out | 0
.../results/postgreSQL/numeric.sql.out | 0
.../results/postgreSQL/numeric.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/select.sql.out | 0
.../results/postgreSQL/select_distinct.sql.out | 0
.../results/postgreSQL/select_having.sql.out | 0
.../results/postgreSQL/select_implicit.sql.out | 0
.../results/postgreSQL/strings.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/text.sql.out | 0
.../results/postgreSQL/timestamp.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/union.sql.out | 0
.../results/postgreSQL/window_part1.sql.out | 0
.../results/postgreSQL/window_part2.sql.out | 0
.../results/postgreSQL/window_part3.sql.out | 0
.../results/postgreSQL/window_part4.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/postgreSQL/with.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/pred-pushdown.sql.out | 0
.../results/predicate-functions.sql.out | 0
.../results/query_regex_column.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/random.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/regexp-functions.sql.out | 0
.../results/replacing-missing-expression-with-alias.sql.out | 0
.../results/runtime-replaceable-edge-cases.sql.out | 0
.../results/scripting/randomly_generated_scripts.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/selectExcept.sql.out | 0
.../results/session-variable-precedence.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show-create-table.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show-tables.sql.out | 0
.../results/show-tblproperties.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show-views.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/show_columns.sql.out | 0
.../results/sql-compatibility-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/sql-on-files.sql.out | 0
.../results/sql-session-variables.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/sql-udf.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/st-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/string-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/struct.sql.out | 0
.../results/subexp-elimination.sql.out | 0
.../results/subquery/exists-subquery/exists-aggregate.sql.out | 0
.../results/subquery/exists-subquery/exists-basic.sql.out | 0
.../results/subquery/exists-subquery/exists-count-bug.sql.out | 0
.../results/subquery/exists-subquery/exists-cte.sql.out | 0
.../results/subquery/exists-subquery/exists-having.sql.out | 0
.../subquery/exists-subquery/exists-in-join-condition.sql.out | 0
.../subquery/exists-subquery/exists-joins-and-set-ops.sql.out | 0
.../results/subquery/exists-subquery/exists-orderby-limit.sql.out | 0
.../results/subquery/exists-subquery/exists-outside-filter.sql.out | 0
.../results/subquery/exists-subquery/exists-within-and-or.sql.out | 0
.../results/subquery/in-subquery/in-basic.sql.out | 0
.../results/subquery/in-subquery/in-count-bug.sql.out | 0
.../results/subquery/in-subquery/in-group-by.sql.out | 0
.../results/subquery/in-subquery/in-having.sql.out | 0
.../results/subquery/in-subquery/in-joins.sql.out | 0
.../results/subquery/in-subquery/in-limit.sql.out | 0
.../results/subquery/in-subquery/in-multiple-columns.sql.out | 0
.../results/subquery/in-subquery/in-null-semantics.sql.out | 0
.../results/subquery/in-subquery/in-nullability.sql.out | 0
.../results/subquery/in-subquery/in-order-by.sql.out | 0
.../results/subquery/in-subquery/in-set-operations.sql.out | 0
.../subquery/in-subquery/in-subquery-in-join-condition.sql.out | 0
.../results/subquery/in-subquery/in-with-cte.sql.out | 0
.../results/subquery/in-subquery/nested-not-in.sql.out | 0
.../results/subquery/in-subquery/not-in-group-by.sql.out | 0
.../results/subquery/in-subquery/not-in-joins.sql.out | 0
.../in-subquery/not-in-unit-tests-multi-column-literal.sql.out | 0
.../subquery/in-subquery/not-in-unit-tests-multi-column.sql.out | 0
.../in-subquery/not-in-unit-tests-single-column-literal.sql.out | 0
.../subquery/in-subquery/not-in-unit-tests-single-column.sql.out | 0
.../results/subquery/in-subquery/simple-in.sql.out | 0
.../results/subquery/negative-cases/invalid-correlation.sql.out | 0
.../results/subquery/negative-cases/subq-input-typecheck.sql.out | 0
.../results/subquery/resolution-edge-cases.sql.out | 0
.../scalar-subquery/nested-scalar-subquery-count-bug.sql.out | 0
.../subquery/scalar-subquery/scalar-subquery-count-bug.sql.out | 0
.../subquery/scalar-subquery/scalar-subquery-group-by.sql.out | 0
.../subquery/scalar-subquery/scalar-subquery-predicate.sql.out | 0
.../results/subquery/scalar-subquery/scalar-subquery-select.sql.out | 0
.../results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out | 0
.../results/subquery/subquery-in-from.sql.out | 0
.../results/subquery/subquery-nested-data.sql.out | 0
.../results/subquery/subquery-offset.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/table-aliases.sql.out | 0
.../results/table-valued-functions.sql.out | 0
.../results/tablesample-negative.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/thetasketch.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/time.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timestamp-ltz.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timestamp-ntz.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timestamp.sql.out | 0
.../results/timestampNTZ/datetime-special-ansi.sql.out | 0
.../results/timestampNTZ/datetime-special.sql.out | 0
.../results/timestampNTZ/timestamp-ansi.sql.out | 0
.../results/timestampNTZ/timestamp.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/timezone.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/to_from_avro.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/transform.sql.out | 0
.../results/try-string-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_aggregates.sql.out | 0
.../results/try_aggregates.sql.out.java21 | 0
.../sql-tests => sql-tests-gluten}/results/try_arithmetic.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_cast.sql.out | 0
.../results/try_datetime_functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_element_at.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/try_reflect.sql.out | 0
.../results/type-coercion-edge-cases.sql.out | 0
.../results/typeCoercion/native/arrayJoin.sql.out | 0
.../results/typeCoercion/native/binaryComparison.sql.out | 0
.../results/typeCoercion/native/booleanEquality.sql.out | 0
.../results/typeCoercion/native/caseWhenCoercion.sql.out | 0
.../results/typeCoercion/native/concat.sql.out | 0
.../results/typeCoercion/native/dateTimeOperations.sql.out | 0
.../results/typeCoercion/native/decimalPrecision.sql.out | 0
.../results/typeCoercion/native/division.sql.out | 0
.../results/typeCoercion/native/elt.sql.out | 0
.../results/typeCoercion/native/ifCoercion.sql.out | 0
.../results/typeCoercion/native/implicitTypeCasts.sql.out | 0
.../results/typeCoercion/native/inConversion.sql.out | 0
.../results/typeCoercion/native/mapZipWith.sql.out | 0
.../results/typeCoercion/native/mapconcat.sql.out | 0
.../results/typeCoercion/native/promoteStrings.sql.out | 0
.../results/typeCoercion/native/stringCastAndExpressions.sql.out | 0
.../results/typeCoercion/native/widenSetOperationTypes.sql.out | 0
.../results/typeCoercion/native/windowFrameCoercion.sql.out | 0
.../results/udaf/udaf-group-analytics.sql.out | 0
.../results/udaf/udaf-group-by-ordinal.sql.out | 0
.../results/udaf/udaf-group-by.sql.out | 0
.../results/udaf/udaf-grouping-set.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udaf/udaf.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part1.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part2.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part3.sql.out | 0
.../results/udf/postgreSQL/udf-aggregates_part4.sql.out | 0
.../results/udf/postgreSQL/udf-case.sql.out | 0
.../results/udf/postgreSQL/udf-join.sql.out | 0
.../results/udf/postgreSQL/udf-select_having.sql.out | 0
.../results/udf/postgreSQL/udf-select_implicit.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-count.sql.out | 0
.../results/udf/udf-cross-join.sql.out | 0
.../results/udf/udf-except-all.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-except.sql.out | 0
.../results/udf/udf-group-analytics.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-group-by.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-having.sql.out | 0
.../results/udf/udf-inline-table.sql.out | 0
.../results/udf/udf-inner-join.sql.out | 0
.../results/udf/udf-intersect-all.sql.out | 0
.../results/udf/udf-join-empty-relation.sql.out | 0
.../results/udf/udf-natural-join.sql.out | 0
.../results/udf/udf-outer-join.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-pivot.sql.out | 0
.../results/udf/udf-special-values.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-udaf.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-union.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udf/udf-window.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/udtf/udtf.sql.out | 0
.../results/union-per-child-output-deduplication.sql.out | 0
.../results/union-resolution-edge-cases.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/union.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/unpivot.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/url-functions.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/using-join.sql.out | 0
.../results/variant-field-extractions.sql.out | 0
.../results/variant/named-function-arguments.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/view-conf.sql.out | 0
.../results/view-schema-binding-config.sql.out | 0
.../results/view-schema-binding.sql.out | 0
.../results/view-schema-compensation.sql.out | 0
.../results/view-schema-evolution.sql.out | 0
.../results/view-schema-type-evolution.sql.out | 0
.../results/view-with-default-collation.sql.out | 0
.../results/window-with-aggregates.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/window.sql.out | 0
.../sql-tests => sql-tests-gluten}/results/xml-functions.sql.out | 0
.../org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | 3 ---
1354 files changed, 4 insertions(+), 10 deletions(-)
diff --git a/gluten-ut/common/src/test/scala/org/apache/gluten/utils/SQLQueryTestSettings.scala b/gluten-ut/common/src/test/scala/org/apache/gluten/utils/SQLQueryTestSettings.scala
index fa88192e23..13f37411cc 100644
--- a/gluten-ut/common/src/test/scala/org/apache/gluten/utils/SQLQueryTestSettings.scala
+++ b/gluten-ut/common/src/test/scala/org/apache/gluten/utils/SQLQueryTestSettings.scala
@@ -17,7 +17,10 @@
package org.apache.gluten.utils
trait SQLQueryTestSettings {
- def getResourceFilePath: String
+ // The backend-neutral, shared directory holding Gluten's overwrite sql-tests
+ // (inputs/results). All backends reuse the same resources to avoid duplication.
+ def getResourceFilePath: String =
+ getClass.getResource("/").getPath + "../../../src/test/resources/sql-tests-gluten"
def getSupportedSQLQueryTests: Set[String]
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/array.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/array.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/array.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/array.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/bitwise.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/bitwise.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/bitwise.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/bitwise.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/charvarchar.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/charvarchar.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/charvarchar.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/charvarchar.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/count.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/count.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/count.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/count.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/cte.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/cte.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/cte.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/cte.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/date.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/date.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/date.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/date.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-legacy.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-legacy.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-legacy.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-legacy.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-invalid.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-invalid.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-invalid.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-invalid.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-legacy.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-legacy.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-legacy.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-legacy.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-parsing.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/datetime-parsing.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/group-by-ordinal.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/group-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/group-by-ordinal.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/group-by-ordinal.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/group-by.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/group-by.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/group-by.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/group-by.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/hll.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/hll.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/hll.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/hll.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/interval.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/interval.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/interval.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/interval.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/linear-regression.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/linear-regression.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/linear-regression.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/linear-regression.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/misc-functions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/misc-functions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/misc-functions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/misc-functions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/percentiles.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/percentiles.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/percentiles.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/percentiles.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part1.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part1.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part1.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part1.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part3.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part3.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part3.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part3.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part4.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part4.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part4.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part4.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/random.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/random.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/random.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/random.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/regexp-functions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/regexp-functions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/regexp-functions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/regexp-functions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/string-functions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/string-functions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/string-functions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/string-functions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-null-semantics.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-null-semantics.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-null-semantics.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-null-semantics.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/table-valued-functions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/table-valued-functions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/table-valued-functions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/table-valued-functions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/timestamp.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/timestamp.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/timestamp.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/timestamp.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_arithmetic.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_arithmetic.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_arithmetic.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_arithmetic.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_cast.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_cast.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_cast.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_cast.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_datetime_functions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_datetime_functions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_datetime_functions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_datetime_functions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_element_at.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_element_at.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/try_element_at.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/try_element_at.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/stringCastAndExpressions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/stringCastAndExpressions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/stringCastAndExpressions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/stringCastAndExpressions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-by.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-by.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-by.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-by.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/url-functions.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/url-functions.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/url-functions.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/url-functions.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/window.sql b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/window.sql
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/inputs/window.sql
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/inputs/window.sql
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/array.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/array.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/array.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/array.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/bitwise.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/bitwise.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/bitwise.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/bitwise.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/charvarchar.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/charvarchar.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/charvarchar.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/charvarchar.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/count.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/count.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/count.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/count.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/cte.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/cte.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/cte.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/cte.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-legacy.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-legacy.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-legacy.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-invalid.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-parsing-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-invalid.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-parsing-invalid.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-legacy.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-parsing-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-legacy.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-parsing-legacy.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-parsing.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-parsing.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/datetime-parsing.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/datetime-parsing.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/group-by-ordinal.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/group-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/group-by-ordinal.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/group-by-ordinal.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/group-by.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/group-by.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/group-by.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/group-by.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/hll.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/hll.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/hll.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/hll.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/interval.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/interval.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/interval.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/interval.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/linear-regression.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/linear-regression.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/linear-regression.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/linear-regression.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/misc-functions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/misc-functions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/misc-functions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/misc-functions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/percentiles.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/percentiles.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/percentiles.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/percentiles.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part1.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part1.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part1.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part1.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part3.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part3.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part3.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part3.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part4.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part4.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part4.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part4.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/random.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/random.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/random.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/random.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/regexp-functions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/regexp-functions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/regexp-functions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/regexp-functions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/string-functions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/string-functions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/string-functions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-null-semantics.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-null-semantics.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-null-semantics.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-null-semantics.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/table-valued-functions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/table-valued-functions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/table-valued-functions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/table-valued-functions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_arithmetic.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_arithmetic.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_arithmetic.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_arithmetic.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_cast.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_cast.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_cast.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_cast.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_datetime_functions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_datetime_functions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_datetime_functions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_datetime_functions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_element_at.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_element_at.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/try_element_at.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/try_element_at.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/typeCoercion/native/stringCastAndExpressions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/typeCoercion/native/stringCastAndExpressions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-by.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/udf/udf-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-by.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/udf/udf-group-by.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/url-functions.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/url-functions.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/url-functions.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/url-functions.sql.out
diff --git a/gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/window.sql.out b/gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/window.sql.out
similarity index 100%
rename from gluten-ut/spark35/src/test/resources/backends-velox/sql-tests/results/window.sql.out
rename to gluten-ut/spark35/src/test/resources/sql-tests-gluten/results/window.sql.out
diff --git a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
index c94a600b3c..2388730d8a 100644
--- a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
+++ b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
@@ -19,9 +19,6 @@ package org.apache.gluten.utils.velox
import org.apache.gluten.utils.SQLQueryTestSettings
object VeloxSQLQueryTestSettings extends SQLQueryTestSettings {
- override def getResourceFilePath: String =
- getClass.getResource("/").getPath + "../../../src/test/resources/backends-velox/sql-tests"
-
override def getSupportedSQLQueryTests: Set[String] = SUPPORTED_SQL_QUERY_LIST
override def getOverwriteSQLQueryTests: Set[String] = OVERWRITE_SQL_QUERY_LIST
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/array.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/array.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/array.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/array.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_base64.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_base64.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_base64.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_base64.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_basic.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_basic.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_basic.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_basic.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_hex.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_hex.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_hex.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_hex.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_hex_discrete.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_hex_discrete.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/binary_hex_discrete.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/binary_hex_discrete.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/bitwise.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/bitwise.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/bitwise.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/bitwise.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cast.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cast.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cast.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cast.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ceil-floor-with-scale-param.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ceil-floor-with-scale-param.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ceil-floor-with-scale-param.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ceil-floor-with-scale-param.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/change-column.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/change-column.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/change-column.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/change-column.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/charvarchar.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/charvarchar.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/charvarchar.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/charvarchar.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/collations.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/collations.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/collations.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/collations.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-aggregate.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/column-resolution-aggregate.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-aggregate.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/column-resolution-aggregate.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-sort.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/column-resolution-sort.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-sort.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/column-resolution-sort.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-negative.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/columnresolution-negative.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-negative.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/columnresolution-negative.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-views.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/columnresolution-views.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-views.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/columnresolution-views.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/columnresolution.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/columnresolution.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/columnresolution.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/columnresolution.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/comments.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/comments.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/comments.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/comments.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/comparator.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/comparator.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/comparator.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/comparator.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/conditional-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/conditional-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/conditional-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/conditional-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/count.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/count.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/count.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/count.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cross-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cross-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cross-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cross-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/csv-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/csv-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/csv-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/csv-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-command.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-command.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-command.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-command.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-legacy.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-legacy.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-legacy.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-legacy.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-nested.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-nested.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-nested.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-nested.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-nonlegacy.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-nonlegacy.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte-nonlegacy.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte-nonlegacy.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/cte.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/cte.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/current_database_catalog.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/current_database_catalog.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/current_database_catalog.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/current_database_catalog.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/date.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/date.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/date.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/date.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-invalid.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-invalid.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-invalid.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-invalid.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-legacy.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-legacy.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-legacy.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-legacy.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-formatting.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-formatting.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-legacy.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-legacy.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-legacy.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-legacy.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-invalid.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-invalid.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-invalid.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-invalid.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-legacy.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-legacy.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-legacy.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-legacy.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-parsing.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-parsing.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-special.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-special.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/datetime-special.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/datetime-special.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/decimalArithmeticOperations.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/decimalArithmeticOperations.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/decimalArithmeticOperations.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/decimalArithmeticOperations.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe-part-after-analyze.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe-part-after-analyze.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe-part-after-analyze.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe-part-after-analyze.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe-query.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe-query.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe-query.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe-query.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe-table-after-alter-table.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe-table-after-alter-table.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe-table-after-alter-table.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe-table-after-alter-table.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/describe.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/describe.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers-enabled.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers-enabled.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers-enabled.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers-enabled.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/except-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/except-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/except-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/except-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/except.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/except.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/except.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/except.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/execute-immediate.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/execute-immediate.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/execute-immediate.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/execute-immediate.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/explain-aqe.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/explain-aqe.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/explain-aqe.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/explain-aqe.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/explain-cbo.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/explain-cbo.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/explain-cbo.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/explain-cbo.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/explain.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/explain.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/explain.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/explain.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/extract.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/extract.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/extract.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/extract.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-analytics.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-analytics.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-analytics.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-analytics.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-alias.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-alias.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-alias.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-alias.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-duckdb.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-all-duckdb.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-duckdb.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-all-duckdb.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-mosha.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-all-mosha.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-mosha.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-all-mosha.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-filter.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-filter.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-filter.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-filter.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-ordinal.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by-ordinal.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by-ordinal.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/group-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/group-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/grouping_set.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/grouping_set.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/grouping_set.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/grouping_set.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/having.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/having.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/having.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/having.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/higher-order-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/higher-order-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/higher-order-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/higher-order-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/hll.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/hll.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/hll.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/hll.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/identifier-clause.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/identifier-clause.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/identifier-clause.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/identifier-clause.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ignored.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ignored.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ignored.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ignored.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ilike-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ilike-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ilike-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ilike-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ilike-any.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ilike-any.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ilike-any.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ilike-any.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ilike.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ilike.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/ilike.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/ilike.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/inline-table.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/inline-table.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/inline-table.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/inline-table.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/inner-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/inner-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/inner-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/inner-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/intersect-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/intersect-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/intersect-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/intersect-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/interval.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/interval.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/interval.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/interval.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/join-empty-relation.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/join-empty-relation.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/join-empty-relation.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/join-empty-relation.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/join-lateral.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/join-lateral.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/join-lateral.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/join-lateral.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/json-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/json-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/json-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/json-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/keywords-enforced.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/keywords-enforced.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/keywords-enforced.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/keywords-enforced.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/keywords.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/keywords.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/keywords.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/keywords.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/like-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/like-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/like-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/like-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/like-any.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/like-any.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/like-any.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/like-any.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/limit.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/limit.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/limit.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/limit.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/linear-regression.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/linear-regression.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/linear-regression.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/linear-regression.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/listagg-collations.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/listagg-collations.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/listagg-collations.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/listagg-collations.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/listagg.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/listagg.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/listagg.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/listagg.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/literals.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/literals.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/literals.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/literals.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/map.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/map.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/map.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/map.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/mask-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/mask-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/mask-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/mask-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/math.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/math.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/math.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/math.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/misc-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/misc-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/misc-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/misc-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/mode.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/mode.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/mode.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/mode.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/named-function-arguments.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/named-function-arguments.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/named-function-arguments.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/named-function-arguments.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/natural-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/natural-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/natural-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/natural-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/non-excludable-rule.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/non-excludable-rule.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/non-excludable-rule.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/non-excludable-rule.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/array.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/array.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/array.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/array.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/cast.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/cast.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/cast.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/cast.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/conditional-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/conditional-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/conditional-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/conditional-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/date.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/date.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/date.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/date.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-parsing-invalid.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-parsing-invalid.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-parsing-invalid.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-parsing-invalid.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-special.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-special.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-special.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-special.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/decimalArithmeticOperations.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/decimalArithmeticOperations.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/decimalArithmeticOperations.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/decimalArithmeticOperations.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/double-quoted-identifiers.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/double-quoted-identifiers.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/double-quoted-identifiers.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/double-quoted-identifiers.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/higher-order-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/higher-order-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/higher-order-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/higher-order-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/interval.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/interval.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/interval.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/interval.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/keywords.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/keywords.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/keywords.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/keywords.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/literals.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/literals.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/literals.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/literals.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/map.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/map.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/map.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/map.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/math.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/math.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/math.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/math.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/parse-schema-string.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/parse-schema-string.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/parse-schema-string.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/parse-schema-string.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/string-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/string-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/string-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/string-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/timestamp.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/timestamp.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/timestamp.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/timestamp.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_aggregates.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_aggregates.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_aggregates.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_aggregates.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_arithmetic.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_arithmetic.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_arithmetic.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_arithmetic.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_datetime_functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_datetime_functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_datetime_functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_datetime_functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_element_at.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_element_at.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_element_at.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/nonansi/try_element_at.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/null-handling.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/null-handling.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/null-handling.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/null-handling.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/null-propagation.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/null-propagation.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/null-propagation.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/null-propagation.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/operators.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/operators.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/operators.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/operators.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by-nulls-ordering.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by-nulls-ordering.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by-nulls-ordering.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by-nulls-ordering.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by-ordinal.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by-ordinal.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by-ordinal.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/order-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/order-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/outer-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/outer-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/outer-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/outer-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness-old-behavior.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness-old-behavior.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness-old-behavior.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness-old-behavior.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/parse-schema-string.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/parse-schema-string.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/parse-schema-string.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/parse-schema-string.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/percentiles.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/percentiles.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/percentiles.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/percentiles.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/pipe-operators.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/pipe-operators.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/pipe-operators.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/pipe-operators.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/pivot.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/pivot.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/pivot.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/pivot.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part1.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part1.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part1.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part1.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part2.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part2.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part2.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part2.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part3.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part3.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part3.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part3.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part4.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part4.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part4.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part4.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/boolean.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/boolean.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/boolean.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/boolean.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/case.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/case.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/case.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/case.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/comments.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/comments.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/comments.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/comments.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/create_view.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/create_view.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/create_view.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/create_view.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/date.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/date.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/date.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/date.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float4.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float4.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float4.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float4.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float8.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float8.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float8.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float8.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/groupingsets.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/groupingsets.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/groupingsets.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/groupingsets.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/insert.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/insert.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/insert.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/insert.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int2.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int2.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int2.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int2.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int4.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int4.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int4.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int4.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int8.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int8.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int8.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int8.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/interval.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/interval.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/interval.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/interval.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/limit.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/limit.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/limit.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/limit.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/numeric.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/numeric.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/numeric.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/numeric.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_distinct.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_distinct.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_distinct.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_distinct.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_having.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_having.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_having.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_having.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_implicit.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_implicit.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_implicit.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_implicit.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/strings.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/strings.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/strings.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/strings.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/text.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/text.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/text.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/text.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/timestamp.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/timestamp.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/timestamp.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/timestamp.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/union.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/union.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/union.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/union.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part1.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part1.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part1.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part1.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part2.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part2.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part2.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part2.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part3.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part3.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part3.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part3.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part4.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part4.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part4.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part4.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/with.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/with.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/with.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/postgreSQL/with.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/pred-pushdown.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/pred-pushdown.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/pred-pushdown.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/pred-pushdown.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/predicate-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/predicate-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/predicate-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/predicate-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/query_regex_column.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/query_regex_column.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/query_regex_column.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/query_regex_column.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/random.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/random.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/random.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/random.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/regexp-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/regexp-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/regexp-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/regexp-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/selectExcept.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/selectExcept.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/selectExcept.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/selectExcept.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-create-table.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-create-table.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-create-table.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-create-table.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-tables.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-tables.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-tables.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-tables.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-tblproperties.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-tblproperties.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-tblproperties.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-tblproperties.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-views.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-views.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show-views.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show-views.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show_columns.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show_columns.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/show_columns.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/show_columns.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-compatibility-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-compatibility-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-compatibility-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-compatibility-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-on-files.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-on-files.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-on-files.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-on-files.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-session-variables.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-session-variables.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-session-variables.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-session-variables.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-udf.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-udf.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/sql-udf.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/sql-udf.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/string-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/string-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/string-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/string-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/struct.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/struct.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/struct.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/struct.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subexp-elimination.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subexp-elimination.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subexp-elimination.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subexp-elimination.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-aggregate.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-aggregate.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-aggregate.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-aggregate.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-basic.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-basic.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-basic.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-basic.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-count-bug.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-count-bug.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-count-bug.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-count-bug.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-cte.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-cte.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-cte.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-cte.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-having.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-having.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-having.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-having.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-in-join-condition.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-in-join-condition.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-in-join-condition.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-in-join-condition.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-orderby-limit.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-orderby-limit.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-orderby-limit.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-orderby-limit.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-outside-filter.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-outside-filter.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-within-and-or.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-within-and-or.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-within-and-or.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-within-and-or.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-basic.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-basic.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-basic.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-basic.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-count-bug.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-count-bug.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-count-bug.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-count-bug.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-group-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-group-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-group-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-group-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-having.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-having.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-having.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-having.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-joins.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-joins.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-joins.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-joins.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-limit.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-limit.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-limit.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-limit.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-multiple-columns.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-multiple-columns.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-multiple-columns.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-multiple-columns.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-null-semantics.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-null-semantics.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-null-semantics.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-null-semantics.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-nullability.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-nullability.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-nullability.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-nullability.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-order-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-order-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-order-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-order-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-set-operations.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-set-operations.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-set-operations.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-set-operations.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-with-cte.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-with-cte.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-with-cte.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-with-cte.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/nested-not-in.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/nested-not-in.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/nested-not-in.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/nested-not-in.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-group-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-group-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-group-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-group-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-joins.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-joins.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/simple-in.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/simple-in.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/simple-in.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/simple-in.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/invalid-correlation.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/invalid-correlation.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/invalid-correlation.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/invalid-correlation.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/subq-input-typecheck.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/subq-input-typecheck.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/subq-input-typecheck.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/subq-input-typecheck.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-select.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-select.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-select.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-select.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-in-from.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-in-from.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-in-from.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-in-from.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-nested-data.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-nested-data.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-nested-data.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-nested-data.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-offset.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-offset.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-offset.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-offset.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/table-aliases.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/table-aliases.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/table-aliases.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/table-aliases.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/table-valued-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/table-valued-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/table-valued-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/table-valued-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/tablesample-negative.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/tablesample-negative.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/tablesample-negative.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/tablesample-negative.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestamp-ltz.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestamp-ltz.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestamp-ltz.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestamp-ltz.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestamp.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestamp.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestamp.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestamp.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special-ansi.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special-ansi.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special-ansi.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special-ansi.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp-ansi.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp-ansi.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp-ansi.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp-ansi.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timezone.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timezone.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/timezone.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/timezone.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/to_from_avro.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/to_from_avro.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/to_from_avro.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/to_from_avro.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/transform.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/transform.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/transform.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/transform.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try-string-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try-string-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try-string-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try-string-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_aggregates.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_aggregates.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_aggregates.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_aggregates.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_arithmetic.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_arithmetic.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_arithmetic.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_arithmetic.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_cast.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_cast.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_cast.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_cast.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_datetime_functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_datetime_functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_datetime_functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_datetime_functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_element_at.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_element_at.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_element_at.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_element_at.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_reflect.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_reflect.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/try_reflect.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/try_reflect.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/arrayJoin.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/arrayJoin.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/arrayJoin.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/arrayJoin.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/binaryComparison.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/binaryComparison.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/binaryComparison.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/binaryComparison.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/booleanEquality.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/booleanEquality.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/booleanEquality.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/booleanEquality.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/caseWhenCoercion.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/caseWhenCoercion.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/caseWhenCoercion.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/caseWhenCoercion.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/concat.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/concat.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/concat.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/concat.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/dateTimeOperations.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/dateTimeOperations.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/dateTimeOperations.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/dateTimeOperations.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/decimalPrecision.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/decimalPrecision.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/decimalPrecision.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/decimalPrecision.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/division.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/division.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/division.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/division.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/elt.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/elt.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/elt.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/elt.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/ifCoercion.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/ifCoercion.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/ifCoercion.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/ifCoercion.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/implicitTypeCasts.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/implicitTypeCasts.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/inConversion.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/inConversion.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/inConversion.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/inConversion.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapZipWith.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapZipWith.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapZipWith.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapZipWith.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapconcat.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapconcat.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapconcat.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapconcat.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/promoteStrings.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/promoteStrings.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/promoteStrings.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/promoteStrings.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/stringCastAndExpressions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/stringCastAndExpressions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/stringCastAndExpressions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/stringCastAndExpressions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/widenSetOperationTypes.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/widenSetOperationTypes.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/widenSetOperationTypes.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/widenSetOperationTypes.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/windowFrameCoercion.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/windowFrameCoercion.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/windowFrameCoercion.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/windowFrameCoercion.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-analytics.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-analytics.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-analytics.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-analytics.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by-ordinal.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by-ordinal.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by-ordinal.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-grouping-set.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-grouping-set.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-grouping-set.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-grouping-set.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udaf/udaf.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part1.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part1.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part1.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part1.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part2.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part2.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part2.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part2.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part3.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part3.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part3.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part3.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part4.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part4.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part4.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part4.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-case.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-case.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-case.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-case.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_having.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_having.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_having.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_having.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_implicit.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_implicit.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_implicit.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_implicit.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-count.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-count.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-count.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-count.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-cross-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-cross-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-cross-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-cross-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-except-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-except-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-except.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-except.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-analytics.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-analytics.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-analytics.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-analytics.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-by.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-by.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-by.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-by.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-having.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-having.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-having.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-having.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inline-table.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-inline-table.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inline-table.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-inline-table.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inner-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-inner-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inner-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-inner-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-intersect-all.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-intersect-all.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-intersect-all.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-intersect-all.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-join-empty-relation.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-join-empty-relation.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-join-empty-relation.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-join-empty-relation.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-natural-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-natural-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-natural-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-natural-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-outer-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-outer-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-outer-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-outer-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-pivot.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-pivot.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-pivot.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-pivot.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-special-values.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-special-values.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-special-values.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-special-values.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-udaf.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-udaf.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-udaf.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-udaf.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-union.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-union.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-union.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-union.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-window.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-window.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-window.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udf/udf-window.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udtf/udtf.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udtf/udtf.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/udtf/udtf.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/udtf/udtf.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/union.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/union.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/union.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/union.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/unpivot.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/unpivot.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/unpivot.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/unpivot.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/url-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/url-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/url-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/url-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/using-join.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/using-join.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/using-join.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/using-join.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/variant/named-function-arguments.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/variant/named-function-arguments.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/variant/named-function-arguments.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/variant/named-function-arguments.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding-config.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-binding-config.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding-config.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-binding-config.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-binding.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-binding.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-compensation.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-compensation.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-compensation.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-compensation.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-evolution.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-evolution.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-evolution.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-evolution.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-type-evolution.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-type-evolution.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-schema-type-evolution.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-schema-type-evolution.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-with-default-collation.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-with-default-collation.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/view-with-default-collation.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/view-with-default-collation.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/window.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/window.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/window.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/window.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/xml-functions.sql b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/xml-functions.sql
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/inputs/xml-functions.sql
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/inputs/xml-functions.sql
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/array.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/array.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/array.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/array.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_base64.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_base64.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_base64.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_base64.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_basic.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_basic.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_basic.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_basic.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_hex.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_hex.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_hex.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_hex.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_hex_discrete.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_hex_discrete.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/binary_hex_discrete.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/binary_hex_discrete.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/bitwise.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/bitwise.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/bitwise.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/bitwise.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cast.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cast.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cast.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cast.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ceil-floor-with-scale-param.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ceil-floor-with-scale-param.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ceil-floor-with-scale-param.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ceil-floor-with-scale-param.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/change-column.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/change-column.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/change-column.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/change-column.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/charvarchar.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/charvarchar.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/charvarchar.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/charvarchar.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/collations.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/collations.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/collations.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/collations.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/column-resolution-aggregate.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/column-resolution-aggregate.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/column-resolution-aggregate.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/column-resolution-aggregate.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/column-resolution-sort.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/column-resolution-sort.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/column-resolution-sort.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/column-resolution-sort.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/columnresolution-negative.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/columnresolution-negative.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/columnresolution-negative.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/columnresolution-negative.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/columnresolution-views.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/columnresolution-views.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/columnresolution-views.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/columnresolution-views.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/columnresolution.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/columnresolution.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/columnresolution.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/columnresolution.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/comments.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/comments.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/comments.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/comments.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/comparator.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/comparator.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/comparator.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/comparator.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/conditional-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/conditional-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/conditional-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/conditional-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/count.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/count.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/count.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/count.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cross-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cross-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cross-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cross-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/csv-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/csv-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/csv-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/csv-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-command.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-command.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-command.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-command.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-legacy.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-legacy.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-legacy.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-nested.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-nested.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-nested.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-nested.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-nonlegacy.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-nonlegacy.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte-nonlegacy.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte-nonlegacy.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/cte.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/cte.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/current_database_catalog.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/current_database_catalog.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/current_database_catalog.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/current_database_catalog.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/date.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/date.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/date.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/date.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-invalid.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-invalid.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting-invalid.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-legacy.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-legacy.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting-legacy.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-legacy.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-legacy.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-legacy.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-invalid.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-parsing-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-invalid.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-parsing-invalid.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-legacy.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-parsing-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-legacy.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-parsing-legacy.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-parsing.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-parsing.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-parsing.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-parsing.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-special.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-special.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/datetime-special.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/datetime-special.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/decimalArithmeticOperations.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/decimalArithmeticOperations.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/decimalArithmeticOperations.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/decimalArithmeticOperations.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe-part-after-analyze.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe-part-after-analyze.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe-part-after-analyze.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe-part-after-analyze.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe-query.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe-query.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe-query.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe-query.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe-table-after-alter-table.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe-table-after-alter-table.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe-table-after-alter-table.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe-table-after-alter-table.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/describe.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/describe.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers-enabled.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers-enabled.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers-enabled.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers-enabled.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/except-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/except-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/except-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/except-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/except.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/except.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/except.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/except.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/execute-immediate.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/execute-immediate.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/execute-immediate.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/execute-immediate.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/explain-aqe.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/explain-aqe.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/explain-aqe.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/explain-aqe.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/explain-cbo.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/explain-cbo.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/explain-cbo.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/explain-cbo.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/explain.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/explain.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/explain.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/explain.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/extract.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/extract.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/extract.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/extract.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-analytics.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-analytics.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-analytics.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-analytics.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-alias.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-alias.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-alias.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-alias.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-all-duckdb.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-all-duckdb.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-all-duckdb.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-all-duckdb.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-all-mosha.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-all-mosha.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-all-mosha.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-all-mosha.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-filter.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-filter.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-filter.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-filter.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-ordinal.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by-ordinal.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by-ordinal.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/group-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/group-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/grouping_set.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/grouping_set.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/grouping_set.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/grouping_set.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/having.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/having.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/having.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/having.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/higher-order-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/higher-order-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/higher-order-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/higher-order-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/hll.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/hll.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/hll.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/hll.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/identifier-clause.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/identifier-clause.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/identifier-clause.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/identifier-clause.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ilike-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ilike-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ilike-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ilike-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ilike-any.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ilike-any.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ilike-any.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ilike-any.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ilike.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ilike.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/ilike.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/ilike.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/inline-table.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/inline-table.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/inline-table.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/inline-table.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/inner-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/inner-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/inner-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/inner-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/intersect-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/intersect-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/intersect-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/intersect-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/interval.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/interval.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/interval.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/interval.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/join-empty-relation.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/join-empty-relation.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/join-empty-relation.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/join-empty-relation.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/join-lateral.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/join-lateral.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/join-lateral.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/join-lateral.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/json-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/json-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/json-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/json-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/keywords-enforced.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/keywords-enforced.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/keywords-enforced.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/keywords-enforced.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/keywords.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/keywords.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/keywords.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/keywords.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/like-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/like-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/like-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/like-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/like-any.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/like-any.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/like-any.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/like-any.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/limit.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/limit.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/limit.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/limit.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/linear-regression.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/linear-regression.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/linear-regression.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/linear-regression.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/listagg-collations.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/listagg-collations.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/listagg-collations.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/listagg-collations.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/listagg.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/listagg.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/listagg.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/listagg.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/literals.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/literals.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/literals.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/literals.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/map.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/map.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/map.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/map.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/mask-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/mask-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/mask-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/mask-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/math.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/math.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/math.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/math.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/misc-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/misc-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/misc-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/misc-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/mode.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/mode.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/mode.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/mode.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/named-function-arguments.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/named-function-arguments.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/named-function-arguments.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/named-function-arguments.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/natural-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/natural-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/natural-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/natural-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/non-excludable-rule.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/non-excludable-rule.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/non-excludable-rule.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/non-excludable-rule.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/array.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/array.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/array.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/array.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/cast.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/cast.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/cast.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/cast.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/conditional-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/conditional-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/conditional-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/conditional-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/date.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/date.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/date.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/date.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-parsing-invalid.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/datetime-parsing-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-parsing-invalid.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/datetime-parsing-invalid.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-special.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/datetime-special.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-special.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/datetime-special.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/decimalArithmeticOperations.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/decimalArithmeticOperations.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/decimalArithmeticOperations.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/decimalArithmeticOperations.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/double-quoted-identifiers.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/double-quoted-identifiers.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/double-quoted-identifiers.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/double-quoted-identifiers.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/higher-order-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/higher-order-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/higher-order-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/higher-order-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/interval.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/interval.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/interval.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/interval.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/keywords.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/keywords.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/keywords.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/keywords.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/literals.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/literals.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/literals.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/literals.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/map.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/map.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/map.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/map.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/math.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/math.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/math.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/math.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/parse-schema-string.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/parse-schema-string.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/parse-schema-string.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/parse-schema-string.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/string-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/string-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/string-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/timestamp.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/timestamp.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/timestamp.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_datetime_functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_datetime_functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_datetime_functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_datetime_functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_element_at.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_element_at.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/nonansi/try_element_at.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/nonansi/try_element_at.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/null-handling.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/null-handling.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/null-handling.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/null-handling.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/null-propagation.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/null-propagation.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/null-propagation.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/null-propagation.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/operators.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/operators.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/operators.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/operators.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by-nulls-ordering.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by-nulls-ordering.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by-nulls-ordering.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by-nulls-ordering.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by-ordinal.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by-ordinal.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by-ordinal.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/order-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/order-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/outer-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/outer-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/outer-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/outer-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness-old-behavior.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/parse-query-correctness-old-behavior.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness-old-behavior.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/parse-query-correctness-old-behavior.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/parse-query-correctness.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/parse-query-correctness.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/parse-schema-string.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/parse-schema-string.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/parse-schema-string.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/parse-schema-string.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/percentiles.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/percentiles.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/percentiles.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/percentiles.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/pipe-operators.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/pipe-operators.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/pipe-operators.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/pipe-operators.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/pivot.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/pivot.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/pivot.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/pivot.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part1.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part1.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part1.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part1.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part2.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part2.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part2.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part2.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part3.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part3.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part3.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part3.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part4.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part4.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part4.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part4.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/boolean.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/boolean.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/boolean.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/boolean.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/case.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/case.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/case.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/case.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/comments.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/comments.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/comments.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/comments.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/create_view.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/create_view.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/create_view.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/create_view.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/date.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/date.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/date.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/date.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float8.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/float8.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float8.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/float8.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/groupingsets.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/groupingsets.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/groupingsets.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/groupingsets.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/insert.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/insert.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/insert.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/insert.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int2.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int2.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int2.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int2.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int4.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int4.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int4.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int4.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/interval.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/interval.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/interval.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/interval.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/limit.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/limit.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/limit.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/limit.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_distinct.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select_distinct.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_distinct.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select_distinct.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_having.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select_having.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_having.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select_having.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_implicit.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select_implicit.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_implicit.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/select_implicit.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/strings.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/strings.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/strings.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/strings.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/text.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/text.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/text.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/text.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/timestamp.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/timestamp.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/timestamp.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/union.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/union.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/union.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/union.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part1.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part1.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part1.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part1.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part2.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part2.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part2.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part2.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part3.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part3.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part3.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part3.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part4.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part4.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part4.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part4.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/with.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/with.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/postgreSQL/with.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/postgreSQL/with.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/pred-pushdown.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/pred-pushdown.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/pred-pushdown.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/pred-pushdown.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/predicate-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/predicate-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/predicate-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/predicate-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/query_regex_column.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/query_regex_column.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/query_regex_column.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/query_regex_column.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/random.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/random.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/random.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/random.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/regexp-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/regexp-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/regexp-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/regexp-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/selectExcept.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/selectExcept.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/selectExcept.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/selectExcept.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-create-table.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-create-table.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-create-table.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-create-table.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-tables.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-tables.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-tables.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-tables.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-tblproperties.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-tblproperties.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-tblproperties.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-tblproperties.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-views.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-views.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show-views.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show-views.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show_columns.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show_columns.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/show_columns.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/show_columns.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-compatibility-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-compatibility-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-compatibility-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-compatibility-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-on-files.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-on-files.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-on-files.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-on-files.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-session-variables.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-session-variables.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-session-variables.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-session-variables.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-udf.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-udf.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/sql-udf.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/sql-udf.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/string-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/string-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/string-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/struct.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/struct.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/struct.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/struct.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subexp-elimination.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subexp-elimination.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subexp-elimination.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subexp-elimination.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-aggregate.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-aggregate.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-aggregate.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-aggregate.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-basic.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-basic.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-basic.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-basic.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-count-bug.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-count-bug.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-count-bug.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-cte.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-cte.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-having.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-having.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-having.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-having.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-in-join-condition.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-in-join-condition.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-in-join-condition.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-in-join-condition.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-orderby-limit.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-orderby-limit.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-orderby-limit.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-orderby-limit.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-outside-filter.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-outside-filter.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-outside-filter.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-outside-filter.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-within-and-or.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-within-and-or.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-within-and-or.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-within-and-or.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-basic.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-basic.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-basic.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-basic.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-count-bug.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-count-bug.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-count-bug.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-group-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-group-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-group-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-having.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-having.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-having.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-having.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-joins.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-joins.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-joins.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-joins.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-limit.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-limit.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-limit.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-limit.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-multiple-columns.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-multiple-columns.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-multiple-columns.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-multiple-columns.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-null-semantics.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-null-semantics.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-null-semantics.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-null-semantics.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-nullability.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-nullability.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-nullability.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-nullability.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-order-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-order-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-order-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-order-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-set-operations.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-set-operations.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-set-operations.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-set-operations.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-with-cte.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-with-cte.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-with-cte.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-with-cte.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/nested-not-in.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/nested-not-in.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/nested-not-in.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/nested-not-in.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-group-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-group-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-group-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-joins.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-joins.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/simple-in.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/simple-in.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/simple-in.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/simple-in.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/invalid-correlation.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/invalid-correlation.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/invalid-correlation.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/invalid-correlation.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/subq-input-typecheck.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/subq-input-typecheck.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/subq-input-typecheck.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/subq-input-typecheck.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-select.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-select.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-select.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-select.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-in-from.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/subquery-in-from.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-in-from.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/subquery-in-from.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-nested-data.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/subquery-nested-data.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-nested-data.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/subquery-nested-data.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-offset.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/subquery-offset.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-offset.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/subquery/subquery-offset.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/table-aliases.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/table-aliases.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/table-aliases.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/table-aliases.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/table-valued-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/table-valued-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/table-valued-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/table-valued-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/tablesample-negative.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/tablesample-negative.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/tablesample-negative.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/tablesample-negative.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestamp-ltz.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestamp-ltz.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestamp-ltz.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestamp-ltz.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestamp-ntz.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestamp-ntz.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestamp-ntz.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestamp-ntz.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestamp.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestamp.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestamp.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special-ansi.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special-ansi.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special-ansi.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special-ansi.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp-ansi.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp-ansi.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp-ansi.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp-ansi.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timezone.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timezone.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/timezone.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/timezone.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/to_from_avro.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/to_from_avro.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/to_from_avro.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/to_from_avro.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/transform.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/transform.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/transform.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/transform.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try-string-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try-string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try-string-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try-string-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out.java21 b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out.java21
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out.java21
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out.java21
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_arithmetic.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_arithmetic.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_arithmetic.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_arithmetic.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_cast.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_cast.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_cast.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_cast.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_datetime_functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_datetime_functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_datetime_functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_datetime_functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_element_at.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_element_at.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_element_at.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_element_at.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_reflect.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_reflect.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/try_reflect.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/try_reflect.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/arrayJoin.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/arrayJoin.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/arrayJoin.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/arrayJoin.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/binaryComparison.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/binaryComparison.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/binaryComparison.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/binaryComparison.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/booleanEquality.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/booleanEquality.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/booleanEquality.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/booleanEquality.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/caseWhenCoercion.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/caseWhenCoercion.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/caseWhenCoercion.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/caseWhenCoercion.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/concat.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/concat.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/concat.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/concat.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/dateTimeOperations.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/dateTimeOperations.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/dateTimeOperations.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/dateTimeOperations.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/decimalPrecision.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/decimalPrecision.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/decimalPrecision.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/decimalPrecision.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/division.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/division.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/division.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/division.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/elt.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/elt.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/elt.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/elt.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/ifCoercion.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/ifCoercion.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/ifCoercion.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/ifCoercion.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/implicitTypeCasts.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/implicitTypeCasts.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/inConversion.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/inConversion.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/inConversion.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/inConversion.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapZipWith.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapZipWith.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapZipWith.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapZipWith.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapconcat.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapconcat.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapconcat.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapconcat.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/promoteStrings.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/promoteStrings.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/promoteStrings.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/promoteStrings.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/stringCastAndExpressions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/stringCastAndExpressions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/widenSetOperationTypes.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/widenSetOperationTypes.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/widenSetOperationTypes.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/widenSetOperationTypes.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/windowFrameCoercion.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/windowFrameCoercion.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/windowFrameCoercion.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/typeCoercion/native/windowFrameCoercion.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-analytics.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-analytics.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-analytics.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-analytics.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by-ordinal.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by-ordinal.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by-ordinal.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-grouping-set.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-grouping-set.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-grouping-set.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf-grouping-set.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udaf/udaf.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udaf/udaf.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part1.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part1.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part1.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part1.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part2.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part2.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part2.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part2.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part3.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part3.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part3.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part3.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part4.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part4.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part4.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part4.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-case.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-case.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-case.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-case.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_having.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_having.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_having.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_having.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_implicit.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_implicit.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_implicit.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_implicit.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-count.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-count.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-count.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-count.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-cross-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-cross-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-cross-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-cross-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-except-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-except-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-except-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-except-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-except.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-except.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-except.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-except.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-analytics.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-group-analytics.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-analytics.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-group-analytics.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-by.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-by.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-group-by.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-having.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-having.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-having.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-having.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-inline-table.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-inline-table.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-inline-table.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-inline-table.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-inner-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-inner-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-inner-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-inner-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-intersect-all.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-intersect-all.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-intersect-all.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-intersect-all.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-join-empty-relation.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-join-empty-relation.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-join-empty-relation.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-join-empty-relation.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-natural-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-natural-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-natural-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-natural-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-outer-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-outer-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-outer-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-outer-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-pivot.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-pivot.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-pivot.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-pivot.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-special-values.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-special-values.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-special-values.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-special-values.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-udaf.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-udaf.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-udaf.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-udaf.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-union.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-union.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-union.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-union.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-window.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-window.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udf/udf-window.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udf/udf-window.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udtf/udtf.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udtf/udtf.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/udtf/udtf.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/udtf/udtf.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/union.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/union.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/union.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/union.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/unpivot.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/unpivot.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/unpivot.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/unpivot.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/url-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/url-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/url-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/url-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/using-join.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/using-join.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/using-join.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/using-join.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/variant/named-function-arguments.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/variant/named-function-arguments.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/variant/named-function-arguments.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/variant/named-function-arguments.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-binding-config.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-binding-config.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-binding-config.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-binding-config.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-binding.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-binding.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-binding.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-binding.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-compensation.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-compensation.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-compensation.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-compensation.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-evolution.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-evolution.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-evolution.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-evolution.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-type-evolution.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-type-evolution.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-schema-type-evolution.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-schema-type-evolution.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-with-default-collation.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-with-default-collation.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/view-with-default-collation.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/view-with-default-collation.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/window.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/window.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/window.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/window.sql.out
diff --git a/gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/xml-functions.sql.out b/gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/xml-functions.sql.out
similarity index 100%
rename from gluten-ut/spark40/src/test/resources/backends-velox/sql-tests/results/xml-functions.sql.out
rename to gluten-ut/spark40/src/test/resources/sql-tests-gluten/results/xml-functions.sql.out
diff --git a/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala b/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
index b085257f70..6f44364cb4 100644
--- a/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
+++ b/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
@@ -19,9 +19,6 @@ package org.apache.gluten.utils.velox
import org.apache.gluten.utils.SQLQueryTestSettings
object VeloxSQLQueryTestSettings extends SQLQueryTestSettings {
- override def getResourceFilePath: String =
- getClass.getResource("/").getPath + "../../../src/test/resources/backends-velox/sql-tests"
-
override def getSupportedSQLQueryTests: Set[String] = SUPPORTED_SQL_QUERY_LIST
override def getOverwriteSQLQueryTests: Set[String] = OVERWRITE_SQL_QUERY_LIST
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/alias-resolution-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/alias-resolution-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/alias-resolution-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/alias-resolution-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/array.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/array.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/array.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/array.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_base64.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_base64.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_base64.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_base64.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_basic.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_basic.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_basic.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_basic.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_hex.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_hex.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_hex.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_hex.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_hex_discrete.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_hex_discrete.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/binary_hex_discrete.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/binary_hex_discrete.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/bitwise.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/bitwise.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/bitwise.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/bitwise.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cast.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cast.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cast.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cast.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ceil-floor-with-scale-param.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ceil-floor-with-scale-param.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ceil-floor-with-scale-param.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ceil-floor-with-scale-param.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/change-column.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/change-column.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/change-column.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/change-column.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/charvarchar.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/charvarchar.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/charvarchar.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/charvarchar.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-aliases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-aliases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-aliases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-aliases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-basic.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-basic.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-basic.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-basic.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-padding-trim.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-padding-trim.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-padding-trim.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-padding-trim.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-string-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-string-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/collations-string-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/collations-string-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-aggregate.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/column-resolution-aggregate.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-aggregate.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/column-resolution-aggregate.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-sort.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/column-resolution-sort.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/column-resolution-sort.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/column-resolution-sort.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-negative.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/columnresolution-negative.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-negative.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/columnresolution-negative.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-views.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/columnresolution-views.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/columnresolution-views.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/columnresolution-views.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/columnresolution.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/columnresolution.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/columnresolution.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/columnresolution.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/comments.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/comments.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/comments.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/comments.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/comparator.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/comparator.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/comparator.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/comparator.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/conditional-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/conditional-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/conditional-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/conditional-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/count.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/count.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/count.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/count.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cross-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cross-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cross-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cross-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/csv-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/csv-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/csv-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/csv-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-command.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-command.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-command.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-command.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-legacy.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-legacy.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-legacy.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-legacy.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-nested.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-nested.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-nested.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-nested.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-nonlegacy.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-nonlegacy.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-nonlegacy.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-nonlegacy.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-recursion.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-recursion.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte-recursion.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte-recursion.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/cte.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/cte.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/current_database_catalog.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/current_database_catalog.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/current_database_catalog.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/current_database_catalog.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/date.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/date.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/date.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/date.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-invalid.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-invalid.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-invalid.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-invalid.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-legacy.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-legacy.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting-legacy.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-formatting-legacy.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-formatting.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-formatting.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-formatting.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-legacy.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-legacy.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-legacy.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-legacy.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-invalid.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-invalid.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-invalid.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-invalid.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-legacy.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-legacy.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing-legacy.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-parsing-legacy.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-parsing.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-parsing.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-parsing.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-special.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-special.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/datetime-special.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/datetime-special.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/decimalArithmeticOperations.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/decimalArithmeticOperations.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/decimalArithmeticOperations.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/decimalArithmeticOperations.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe-part-after-analyze.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe-part-after-analyze.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe-part-after-analyze.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe-part-after-analyze.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe-query.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe-query.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe-query.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe-query.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe-table-after-alter-table.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe-table-after-alter-table.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe-table-after-alter-table.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe-table-after-alter-table.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/describe.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/describe.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers-enabled.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers-enabled.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers-enabled.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers-enabled.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/double-quoted-identifiers.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/double-quoted-identifiers.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/except-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/except-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/except-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/except-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/except.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/except.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/except.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/except.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/execute-immediate.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/execute-immediate.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/execute-immediate.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/execute-immediate.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/explain-aqe.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/explain-aqe.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/explain-aqe.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/explain-aqe.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/explain-cbo.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/explain-cbo.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/explain-cbo.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/explain-cbo.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/explain.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/explain.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/explain.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/explain.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/extract-value-resolution-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/extract-value-resolution-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/extract-value-resolution-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/extract-value-resolution-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/extract.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/extract.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/extract.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/extract.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/generators.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/generators.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/generators.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/generators.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-analytics.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-analytics.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-analytics.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-analytics.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-alias.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-alias.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-alias.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-alias.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-duckdb.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-all-duckdb.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-duckdb.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-all-duckdb.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-mosha.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-all-mosha.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-all-mosha.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-all-mosha.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-filter.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-filter.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-filter.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-filter.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-ordinal.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by-ordinal.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by-ordinal.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/group-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/group-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/grouping_set.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/grouping_set.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/grouping_set.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/grouping_set.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/having-and-order-by-recursive-type-name-resolution.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/having-and-order-by-recursive-type-name-resolution.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/having-and-order-by-recursive-type-name-resolution.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/having-and-order-by-recursive-type-name-resolution.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/having.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/having.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/having.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/having.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/higher-order-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/higher-order-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/higher-order-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/higher-order-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/hll.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/hll.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/hll.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/hll.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/identifier-clause-legacy.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/identifier-clause-legacy.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/identifier-clause-legacy.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/identifier-clause-legacy.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/identifier-clause.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/identifier-clause.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/identifier-clause.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/identifier-clause.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ignored.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ignored.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ignored.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ignored.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ilike-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ilike-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ilike-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ilike-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ilike-any.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ilike-any.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ilike-any.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ilike-any.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ilike.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ilike.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/ilike.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/ilike.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/inline-table.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/inline-table.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/inline-table.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/inline-table.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/inner-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/inner-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/inner-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/inner-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/intersect-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/intersect-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/intersect-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/intersect-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/interval.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/interval.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/interval.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/interval.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/join-empty-relation.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/join-empty-relation.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/join-empty-relation.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/join-empty-relation.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/join-lateral.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/join-lateral.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/join-lateral.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/join-lateral.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/join-resolution-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/join-resolution-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/join-resolution-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/join-resolution-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/json-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/json-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/json-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/json-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/keywords-enforced.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/keywords-enforced.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/keywords-enforced.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/keywords-enforced.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/keywords.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/keywords.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/keywords.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/keywords.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/kllquantiles.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/kllquantiles.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/kllquantiles.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/kllquantiles.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/like-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/like-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/like-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/like-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/like-any.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/like-any.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/like-any.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/like-any.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/limit.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/limit.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/limit.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/limit.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/linear-regression.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/linear-regression.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/linear-regression.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/linear-regression.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/listagg-collations.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/listagg-collations.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/listagg-collations.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/listagg-collations.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/listagg.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/listagg.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/listagg.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/listagg.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/literals.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/literals.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/literals.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/literals.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/map.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/map.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/map.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/map.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/mask-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/mask-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/mask-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/mask-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/math.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/math.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/math.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/math.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/misc-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/misc-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/misc-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/misc-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/mode.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/mode.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/mode.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/mode.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/named-function-arguments.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/named-function-arguments.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/named-function-arguments.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/named-function-arguments.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/natural-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/natural-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/natural-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/natural-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/non-excludable-rule.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/non-excludable-rule.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/non-excludable-rule.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/non-excludable-rule.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/array.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/array.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/array.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/array.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/cast.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/cast.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/cast.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/cast.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/conditional-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/conditional-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/conditional-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/conditional-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/date.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/date.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/date.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/date.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-parsing-invalid.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-parsing-invalid.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-parsing-invalid.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-parsing-invalid.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-special.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-special.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/datetime-special.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/datetime-special.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/decimalArithmeticOperations.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/decimalArithmeticOperations.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/decimalArithmeticOperations.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/decimalArithmeticOperations.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/double-quoted-identifiers.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/double-quoted-identifiers.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/double-quoted-identifiers.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/double-quoted-identifiers.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/higher-order-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/higher-order-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/higher-order-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/higher-order-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/interval.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/interval.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/interval.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/interval.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/keywords.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/keywords.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/keywords.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/keywords.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/literals.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/literals.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/literals.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/literals.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/map.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/map.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/map.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/map.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/math.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/math.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/math.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/math.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/parse-schema-string.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/parse-schema-string.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/parse-schema-string.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/parse-schema-string.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/st-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/st-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/st-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/st-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/string-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/string-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/string-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/string-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/timestamp.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/timestamp.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/timestamp.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/timestamp.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_aggregates.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_aggregates.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_aggregates.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_aggregates.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_arithmetic.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_arithmetic.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_arithmetic.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_arithmetic.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_datetime_functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_datetime_functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_datetime_functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_datetime_functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_element_at.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_element_at.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/nonansi/try_element_at.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/nonansi/try_element_at.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/null-handling.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/null-handling.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/null-handling.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/null-handling.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/null-propagation.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/null-propagation.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/null-propagation.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/null-propagation.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/operators.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/operators.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/operators.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/operators.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-alias.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-alias.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-alias.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-alias.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-and-having-on-top-of-aggregate-with-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-and-having-on-top-of-aggregate-with-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-and-having-on-top-of-aggregate-with-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-and-having-on-top-of-aggregate-with-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-nulls-ordering.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-nulls-ordering.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-nulls-ordering.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-nulls-ordering.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-ordinal.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by-ordinal.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by-ordinal.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/order-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/order-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/outer-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/outer-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/outer-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/outer-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness-old-behavior.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness-old-behavior.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness-old-behavior.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness-old-behavior.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/parse-query-correctness.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/parse-query-correctness.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/parse-schema-string.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/parse-schema-string.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/parse-schema-string.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/parse-schema-string.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/percentiles.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/percentiles.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/percentiles.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/percentiles.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/pipe-operators.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/pipe-operators.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/pipe-operators.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/pipe-operators.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/pivot.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/pivot.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/pivot.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/pivot.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part1.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part1.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part1.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part1.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part2.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part2.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part2.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part2.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part3.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part3.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part3.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part3.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part4.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part4.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/aggregates_part4.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/aggregates_part4.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/boolean.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/boolean.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/boolean.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/boolean.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/case.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/case.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/case.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/case.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/comments.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/comments.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/comments.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/comments.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/create_view.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/create_view.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/create_view.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/create_view.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/date.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/date.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/date.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/date.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float4.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float4.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float4.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float4.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float8.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float8.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/float8.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/float8.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/groupingsets.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/groupingsets.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/groupingsets.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/groupingsets.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/insert.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/insert.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/insert.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/insert.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int2.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int2.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int2.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int2.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int4.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int4.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int4.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int4.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int8.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int8.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/int8.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/int8.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/interval.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/interval.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/interval.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/interval.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/limit.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/limit.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/limit.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/limit.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/numeric.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/numeric.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/numeric.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/numeric.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_distinct.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_distinct.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_distinct.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_distinct.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_having.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_having.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_having.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_having.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_implicit.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_implicit.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/select_implicit.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/select_implicit.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/strings.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/strings.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/strings.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/strings.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/text.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/text.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/text.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/text.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/timestamp.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/timestamp.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/timestamp.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/timestamp.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/union.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/union.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/union.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/union.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part1.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part1.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part1.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part1.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part2.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part2.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part2.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part2.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part3.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part3.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part3.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part3.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part4.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part4.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/window_part4.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/window_part4.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/with.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/with.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/postgreSQL/with.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/postgreSQL/with.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/pred-pushdown.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/pred-pushdown.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/pred-pushdown.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/pred-pushdown.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/predicate-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/predicate-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/predicate-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/predicate-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/query_regex_column.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/query_regex_column.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/query_regex_column.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/query_regex_column.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/random.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/random.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/random.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/random.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/regexp-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/regexp-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/regexp-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/regexp-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/replacing-missing-expression-with-alias.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/replacing-missing-expression-with-alias.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/replacing-missing-expression-with-alias.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/replacing-missing-expression-with-alias.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/runtime-replaceable-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/runtime-replaceable-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/runtime-replaceable-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/runtime-replaceable-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/scripting/randomly_generated_scripts.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/scripting/randomly_generated_scripts.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/scripting/randomly_generated_scripts.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/scripting/randomly_generated_scripts.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/selectExcept.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/selectExcept.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/selectExcept.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/selectExcept.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/session-variable-precedence.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/session-variable-precedence.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/session-variable-precedence.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/session-variable-precedence.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-create-table.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-create-table.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-create-table.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-create-table.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-tables.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-tables.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-tables.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-tables.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-tblproperties.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-tblproperties.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-tblproperties.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-tblproperties.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-views.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-views.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show-views.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show-views.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show_columns.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show_columns.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/show_columns.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/show_columns.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-compatibility-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-compatibility-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-compatibility-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-compatibility-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-on-files.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-on-files.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-on-files.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-on-files.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-session-variables.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-session-variables.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-session-variables.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-session-variables.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-udf.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-udf.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/sql-udf.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/sql-udf.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/st-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/st-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/st-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/st-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/string-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/string-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/string-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/string-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/struct.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/struct.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/struct.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/struct.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subexp-elimination.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subexp-elimination.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subexp-elimination.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subexp-elimination.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-aggregate.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-aggregate.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-aggregate.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-aggregate.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-basic.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-basic.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-basic.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-basic.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-count-bug.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-count-bug.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-count-bug.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-count-bug.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-cte.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-cte.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-cte.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-cte.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-having.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-having.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-having.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-having.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-in-join-condition.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-in-join-condition.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-in-join-condition.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-in-join-condition.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-joins-and-set-ops.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-orderby-limit.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-orderby-limit.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-orderby-limit.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-orderby-limit.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-outside-filter.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-outside-filter.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-outside-filter.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-within-and-or.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-within-and-or.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/exists-subquery/exists-within-and-or.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/exists-subquery/exists-within-and-or.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-basic.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-basic.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-basic.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-basic.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-count-bug.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-count-bug.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-count-bug.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-count-bug.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-group-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-group-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-group-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-group-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-having.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-having.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-having.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-having.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-joins.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-joins.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-joins.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-joins.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-limit.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-limit.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-limit.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-limit.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-multiple-columns.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-multiple-columns.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-multiple-columns.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-multiple-columns.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-null-semantics.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-null-semantics.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-null-semantics.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-null-semantics.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-nullability.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-nullability.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-nullability.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-nullability.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-order-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-order-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-order-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-order-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-set-operations.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-set-operations.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-set-operations.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-set-operations.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-subquery-in-join-condition.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-with-cte.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-with-cte.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/in-with-cte.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/in-with-cte.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/nested-not-in.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/nested-not-in.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/nested-not-in.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/nested-not-in.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-group-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-group-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-group-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-group-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-joins.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-joins.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/simple-in.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/simple-in.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/in-subquery/simple-in.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/in-subquery/simple-in.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/invalid-correlation.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/invalid-correlation.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/invalid-correlation.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/invalid-correlation.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/subq-input-typecheck.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/subq-input-typecheck.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/negative-cases/subq-input-typecheck.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/negative-cases/subq-input-typecheck.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/resolution-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/resolution-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/resolution-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/resolution-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-count-bug.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-group-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-predicate.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-select.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-select.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-select.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-select.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/scalar-subquery/scalar-subquery-set-op.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-in-from.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-in-from.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-in-from.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-in-from.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-nested-data.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-nested-data.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-nested-data.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-nested-data.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-offset.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-offset.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/subquery/subquery-offset.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/subquery/subquery-offset.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/table-aliases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/table-aliases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/table-aliases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/table-aliases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/table-valued-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/table-valued-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/table-valued-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/table-valued-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/tablesample-negative.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/tablesample-negative.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/tablesample-negative.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/tablesample-negative.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/thetasketch.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/thetasketch.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/thetasketch.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/thetasketch.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/time.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/time.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/time.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/time.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestamp-ltz.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestamp-ltz.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestamp-ltz.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestamp-ltz.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestamp.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestamp.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestamp.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestamp.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special-ansi.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special-ansi.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special-ansi.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special-ansi.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/datetime-special.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/datetime-special.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp-ansi.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp-ansi.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp-ansi.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp-ansi.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timestampNTZ/timestamp.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timestampNTZ/timestamp.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timezone.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timezone.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/timezone.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/timezone.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/to_from_avro.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/to_from_avro.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/to_from_avro.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/to_from_avro.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/transform.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/transform.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/transform.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/transform.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try-string-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try-string-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try-string-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try-string-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_aggregates.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_aggregates.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_aggregates.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_aggregates.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_arithmetic.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_arithmetic.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_arithmetic.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_arithmetic.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_cast.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_cast.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_cast.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_cast.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_datetime_functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_datetime_functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_datetime_functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_datetime_functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_element_at.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_element_at.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_element_at.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_element_at.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_reflect.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_reflect.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/try_reflect.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/try_reflect.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/type-coercion-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/type-coercion-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/type-coercion-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/type-coercion-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/arrayJoin.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/arrayJoin.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/arrayJoin.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/arrayJoin.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/binaryComparison.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/binaryComparison.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/binaryComparison.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/binaryComparison.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/booleanEquality.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/booleanEquality.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/booleanEquality.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/booleanEquality.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/caseWhenCoercion.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/caseWhenCoercion.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/caseWhenCoercion.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/caseWhenCoercion.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/concat.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/concat.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/concat.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/concat.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/dateTimeOperations.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/dateTimeOperations.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/dateTimeOperations.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/dateTimeOperations.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/decimalPrecision.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/decimalPrecision.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/decimalPrecision.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/decimalPrecision.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/division.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/division.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/division.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/division.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/elt.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/elt.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/elt.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/elt.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/ifCoercion.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/ifCoercion.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/ifCoercion.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/ifCoercion.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/implicitTypeCasts.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/implicitTypeCasts.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/implicitTypeCasts.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/inConversion.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/inConversion.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/inConversion.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/inConversion.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapZipWith.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapZipWith.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapZipWith.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapZipWith.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapconcat.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapconcat.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/mapconcat.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/mapconcat.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/promoteStrings.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/promoteStrings.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/promoteStrings.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/promoteStrings.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/stringCastAndExpressions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/stringCastAndExpressions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/stringCastAndExpressions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/stringCastAndExpressions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/widenSetOperationTypes.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/widenSetOperationTypes.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/widenSetOperationTypes.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/widenSetOperationTypes.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/windowFrameCoercion.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/windowFrameCoercion.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/typeCoercion/native/windowFrameCoercion.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/typeCoercion/native/windowFrameCoercion.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-analytics.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-analytics.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-analytics.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-analytics.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by-ordinal.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by-ordinal.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by-ordinal.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by-ordinal.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-group-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-group-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-grouping-set.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-grouping-set.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf-grouping-set.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf-grouping-set.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udaf/udaf.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udaf/udaf.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part1.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part1.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part1.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part1.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part2.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part2.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part2.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part2.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part3.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part3.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part3.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part3.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part4.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part4.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-aggregates_part4.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-aggregates_part4.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-case.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-case.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-case.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-case.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_having.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_having.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_having.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_having.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_implicit.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_implicit.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/postgreSQL/udf-select_implicit.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/postgreSQL/udf-select_implicit.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-count.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-count.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-count.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-count.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-cross-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-cross-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-cross-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-cross-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-except-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-except-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-except.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-except.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-except.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-analytics.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-analytics.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-analytics.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-analytics.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-by.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-by.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-group-by.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-group-by.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-having.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-having.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-having.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-having.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inline-table.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-inline-table.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inline-table.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-inline-table.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inner-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-inner-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-inner-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-inner-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-intersect-all.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-intersect-all.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-intersect-all.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-intersect-all.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-join-empty-relation.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-join-empty-relation.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-join-empty-relation.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-join-empty-relation.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-natural-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-natural-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-natural-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-natural-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-outer-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-outer-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-outer-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-outer-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-pivot.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-pivot.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-pivot.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-pivot.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-special-values.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-special-values.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-special-values.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-special-values.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-udaf.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-udaf.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-udaf.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-udaf.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-union.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-union.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-union.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-union.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-window.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-window.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udf/udf-window.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udf/udf-window.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udtf/udtf.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udtf/udtf.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/udtf/udtf.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/udtf/udtf.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/union-per-child-output-deduplication.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/union-per-child-output-deduplication.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/union-per-child-output-deduplication.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/union-per-child-output-deduplication.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/union-resolution-edge-cases.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/union-resolution-edge-cases.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/union-resolution-edge-cases.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/union-resolution-edge-cases.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/union.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/union.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/union.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/union.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/unpivot.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/unpivot.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/unpivot.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/unpivot.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/url-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/url-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/url-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/url-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/using-join.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/using-join.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/using-join.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/using-join.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/variant-field-extractions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/variant-field-extractions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/variant-field-extractions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/variant-field-extractions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/variant/named-function-arguments.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/variant/named-function-arguments.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/variant/named-function-arguments.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/variant/named-function-arguments.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-conf.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-conf.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-conf.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-conf.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding-config.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-binding-config.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding-config.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-binding-config.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-binding.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-binding.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-binding.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-compensation.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-compensation.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-compensation.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-compensation.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-evolution.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-evolution.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-evolution.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-evolution.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-type-evolution.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-type-evolution.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-schema-type-evolution.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-schema-type-evolution.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-with-default-collation.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-with-default-collation.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/view-with-default-collation.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/view-with-default-collation.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/window-with-aggregates.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/window-with-aggregates.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/window-with-aggregates.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/window-with-aggregates.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/window.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/window.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/window.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/window.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/xml-functions.sql b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/xml-functions.sql
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/inputs/xml-functions.sql
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/inputs/xml-functions.sql
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/alias-resolution-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/alias-resolution-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/alias-resolution-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/alias-resolution-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/array.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/array.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/array.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/array.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_base64.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_base64.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_base64.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_base64.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_basic.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_basic.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_basic.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_basic.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_hex.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_hex.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_hex.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_hex.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_hex_discrete.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_hex_discrete.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/binary_hex_discrete.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/binary_hex_discrete.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/bitwise.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/bitwise.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/bitwise.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/bitwise.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cast.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cast.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cast.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cast.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ceil-floor-with-scale-param.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ceil-floor-with-scale-param.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ceil-floor-with-scale-param.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ceil-floor-with-scale-param.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/change-column.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/change-column.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/change-column.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/change-column.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/charvarchar.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/charvarchar.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/charvarchar.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/charvarchar.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-aliases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-aliases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-aliases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-aliases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-basic.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-basic.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-basic.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-basic.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-padding-trim.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-padding-trim.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-padding-trim.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-padding-trim.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-string-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/collations-string-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/collations-string-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/column-resolution-aggregate.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/column-resolution-aggregate.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/column-resolution-aggregate.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/column-resolution-aggregate.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/column-resolution-sort.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/column-resolution-sort.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/column-resolution-sort.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/column-resolution-sort.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/columnresolution-negative.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/columnresolution-negative.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/columnresolution-negative.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/columnresolution-negative.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/columnresolution-views.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/columnresolution-views.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/columnresolution-views.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/columnresolution-views.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/columnresolution.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/columnresolution.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/columnresolution.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/columnresolution.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/comments.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/comments.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/comments.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/comments.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/comparator.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/comparator.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/comparator.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/comparator.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/conditional-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/conditional-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/conditional-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/conditional-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/count.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/count.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/count.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/count.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cross-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cross-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cross-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cross-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/csv-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/csv-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/csv-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/csv-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-command.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-command.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-command.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-command.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-legacy.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-legacy.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-legacy.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-nested.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-nested.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-nested.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-nested.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-nonlegacy.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-nonlegacy.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-nonlegacy.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-nonlegacy.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-recursion.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-recursion.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte-recursion.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte-recursion.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/cte.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/cte.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/current_database_catalog.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/current_database_catalog.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/current_database_catalog.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/current_database_catalog.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/date.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/date.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/date.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/date.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-invalid.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-invalid.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting-invalid.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-legacy.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting-legacy.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting-legacy.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-formatting.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-formatting.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-legacy.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-legacy.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-legacy.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-invalid.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-parsing-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-invalid.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-parsing-invalid.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-legacy.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-parsing-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-parsing-legacy.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-parsing-legacy.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-parsing.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-parsing.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-parsing.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-parsing.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-special.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-special.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/datetime-special.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/datetime-special.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/decimalArithmeticOperations.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/decimalArithmeticOperations.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/decimalArithmeticOperations.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/decimalArithmeticOperations.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe-part-after-analyze.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe-part-after-analyze.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe-part-after-analyze.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe-part-after-analyze.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe-query.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe-query.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe-query.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe-query.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe-table-after-alter-table.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe-table-after-alter-table.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe-table-after-alter-table.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe-table-after-alter-table.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/describe.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/describe.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers-enabled.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers-enabled.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers-enabled.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers-enabled.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/double-quoted-identifiers.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/double-quoted-identifiers.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/except-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/except-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/except-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/except-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/except.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/except.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/except.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/except.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/execute-immediate.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/execute-immediate.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/execute-immediate.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/execute-immediate.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/explain-aqe.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/explain-aqe.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/explain-aqe.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/explain-aqe.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/explain-cbo.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/explain-cbo.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/explain-cbo.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/explain-cbo.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/explain.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/explain.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/explain.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/explain.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/extract-value-resolution-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/extract-value-resolution-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/extract-value-resolution-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/extract-value-resolution-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/extract.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/extract.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/extract.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/extract.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/generators.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/generators.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/generators.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/generators.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-analytics.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-analytics.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-analytics.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-analytics.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-alias.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-alias.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-alias.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-alias.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-all-duckdb.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-all-duckdb.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-all-duckdb.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-all-duckdb.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-all-mosha.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-all-mosha.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-all-mosha.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-all-mosha.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-filter.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-filter.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-filter.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-filter.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-ordinal.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by-ordinal.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by-ordinal.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/group-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/group-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/grouping_set.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/grouping_set.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/grouping_set.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/grouping_set.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/having-and-order-by-recursive-type-name-resolution.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/having-and-order-by-recursive-type-name-resolution.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/having-and-order-by-recursive-type-name-resolution.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/having-and-order-by-recursive-type-name-resolution.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/having.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/having.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/having.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/having.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/higher-order-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/higher-order-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/higher-order-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/higher-order-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/hll.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/hll.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/hll.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/hll.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/identifier-clause-legacy.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/identifier-clause-legacy.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/identifier-clause-legacy.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/identifier-clause-legacy.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/identifier-clause.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/identifier-clause.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/identifier-clause.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/identifier-clause.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ilike-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ilike-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ilike-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ilike-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ilike-any.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ilike-any.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ilike-any.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ilike-any.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ilike.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ilike.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/ilike.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/ilike.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/inline-table.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/inline-table.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/inline-table.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/inline-table.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/inner-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/inner-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/inner-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/inner-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/intersect-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/intersect-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/intersect-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/intersect-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/interval.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/interval.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/interval.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/interval.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/join-empty-relation.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/join-empty-relation.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/join-empty-relation.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/join-empty-relation.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/join-lateral.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/join-lateral.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/join-lateral.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/join-lateral.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/join-resolution-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/join-resolution-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/join-resolution-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/join-resolution-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/json-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/json-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/json-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/json-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/keywords-enforced.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/keywords-enforced.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/keywords-enforced.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/keywords-enforced.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/keywords.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/keywords.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/keywords.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/keywords.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/kllquantiles.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/kllquantiles.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/kllquantiles.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/kllquantiles.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/like-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/like-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/like-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/like-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/like-any.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/like-any.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/like-any.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/like-any.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/limit.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/limit.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/limit.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/limit.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/linear-regression.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/linear-regression.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/linear-regression.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/linear-regression.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/listagg-collations.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/listagg-collations.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/listagg-collations.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/listagg-collations.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/listagg.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/listagg.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/listagg.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/listagg.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/literals.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/literals.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/literals.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/literals.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/map.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/map.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/map.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/map.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/mask-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/mask-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/mask-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/mask-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/math.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/math.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/math.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/math.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/misc-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/misc-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/misc-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/misc-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/mode.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/mode.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/mode.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/mode.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/name-precedence-in-order-by-and-having-with-conflicting-attributes.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/named-function-arguments.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/named-function-arguments.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/named-function-arguments.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/named-function-arguments.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/natural-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/natural-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/natural-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/natural-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/non-excludable-rule.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/non-excludable-rule.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/non-excludable-rule.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/non-excludable-rule.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/array.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/array.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/array.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/array.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/cast.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/cast.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/cast.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/cast.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/conditional-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/conditional-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/conditional-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/conditional-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/date.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/date.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/date.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/date.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-parsing-invalid.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/datetime-parsing-invalid.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-parsing-invalid.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/datetime-parsing-invalid.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-special.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/datetime-special.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/datetime-special.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/datetime-special.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/decimalArithmeticOperations.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/decimalArithmeticOperations.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/decimalArithmeticOperations.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/decimalArithmeticOperations.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/double-quoted-identifiers.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/double-quoted-identifiers.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/double-quoted-identifiers.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/double-quoted-identifiers.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/higher-order-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/higher-order-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/higher-order-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/higher-order-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/interval.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/interval.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/interval.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/interval.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/keywords.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/keywords.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/keywords.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/keywords.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/literals.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/literals.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/literals.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/literals.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/map.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/map.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/map.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/map.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/math.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/math.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/math.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/math.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/parse-schema-string.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/parse-schema-string.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/parse-schema-string.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/parse-schema-string.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/st-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/st-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/st-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/st-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/string-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/string-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/string-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/timestamp.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/timestamp.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/timestamp.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_aggregates.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_aggregates.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_arithmetic.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_arithmetic.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_datetime_functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_datetime_functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_datetime_functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_datetime_functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_element_at.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_element_at.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/nonansi/try_element_at.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/nonansi/try_element_at.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/null-handling.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/null-handling.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/null-handling.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/null-handling.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/null-propagation.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/null-propagation.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/null-propagation.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/null-propagation.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/operators.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/operators.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/operators.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/operators.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-alias.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-alias.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-alias.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-alias.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-and-having-on-top-of-aggregate-with-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-and-having-on-top-of-aggregate-with-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-and-having-on-top-of-aggregate-with-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-and-having-on-top-of-aggregate-with-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-nulls-ordering.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-nulls-ordering.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-nulls-ordering.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-nulls-ordering.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-ordinal.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by-ordinal.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by-ordinal.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/order-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/order-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/outer-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/outer-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/outer-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/outer-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness-old-behavior.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/parse-query-correctness-old-behavior.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness-old-behavior.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/parse-query-correctness-old-behavior.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/parse-query-correctness.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/parse-query-correctness.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/parse-query-correctness.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/parse-schema-string.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/parse-schema-string.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/parse-schema-string.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/parse-schema-string.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/percentiles.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/percentiles.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/percentiles.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/percentiles.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/pipe-operators.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/pipe-operators.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/pipe-operators.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/pipe-operators.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/pivot.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/pivot.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/pivot.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/pivot.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part1.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part1.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part1.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part1.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part2.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part2.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part2.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part2.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part3.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part3.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part3.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part3.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part4.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part4.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/aggregates_part4.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/aggregates_part4.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/boolean.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/boolean.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/boolean.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/boolean.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/case.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/case.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/case.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/case.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/comments.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/comments.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/comments.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/comments.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/create_view.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/create_view.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/create_view.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/create_view.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/date.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/date.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/date.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/date.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float4.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/float4.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float8.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/float8.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/float8.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/float8.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/groupingsets.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/groupingsets.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/groupingsets.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/groupingsets.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/insert.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/insert.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/insert.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/insert.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int2.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int2.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int2.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int2.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int4.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int4.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int4.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int4.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/int8.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/int8.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/interval.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/interval.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/interval.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/interval.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/limit.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/limit.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/limit.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/limit.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/numeric.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/numeric.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_distinct.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select_distinct.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_distinct.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select_distinct.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_having.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select_having.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_having.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select_having.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_implicit.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select_implicit.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/select_implicit.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/select_implicit.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/strings.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/strings.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/strings.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/strings.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/text.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/text.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/text.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/text.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/timestamp.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/timestamp.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/timestamp.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/union.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/union.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/union.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/union.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part1.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part1.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part1.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part1.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part2.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part2.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part2.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part2.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part3.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part3.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part3.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part3.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part4.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part4.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/window_part4.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/window_part4.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/with.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/with.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/postgreSQL/with.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/postgreSQL/with.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/pred-pushdown.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/pred-pushdown.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/pred-pushdown.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/pred-pushdown.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/predicate-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/predicate-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/predicate-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/predicate-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/query_regex_column.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/query_regex_column.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/query_regex_column.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/query_regex_column.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/random.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/random.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/random.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/random.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/regexp-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/regexp-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/regexp-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/regexp-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/replacing-missing-expression-with-alias.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/replacing-missing-expression-with-alias.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/replacing-missing-expression-with-alias.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/replacing-missing-expression-with-alias.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/runtime-replaceable-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/runtime-replaceable-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/runtime-replaceable-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/runtime-replaceable-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/scripting/randomly_generated_scripts.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/scripting/randomly_generated_scripts.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/scripting/randomly_generated_scripts.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/scripting/randomly_generated_scripts.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/selectExcept.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/selectExcept.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/selectExcept.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/selectExcept.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/session-variable-precedence.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/session-variable-precedence.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/session-variable-precedence.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/session-variable-precedence.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-create-table.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-create-table.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-create-table.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-create-table.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-tables.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-tables.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-tables.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-tables.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-tblproperties.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-tblproperties.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-tblproperties.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-tblproperties.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-views.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-views.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show-views.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show-views.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show_columns.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show_columns.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/show_columns.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/show_columns.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-compatibility-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-compatibility-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-compatibility-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-compatibility-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-on-files.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-on-files.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-on-files.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-on-files.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-session-variables.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-session-variables.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-session-variables.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-session-variables.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-udf.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-udf.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/sql-udf.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/sql-udf.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/st-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/st-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/st-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/st-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/string-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/string-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/string-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/struct.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/struct.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/struct.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/struct.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subexp-elimination.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subexp-elimination.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subexp-elimination.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subexp-elimination.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-aggregate.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-aggregate.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-aggregate.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-aggregate.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-basic.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-basic.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-basic.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-basic.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-count-bug.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-count-bug.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-count-bug.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-cte.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-cte.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-having.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-having.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-having.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-having.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-in-join-condition.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-in-join-condition.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-in-join-condition.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-in-join-condition.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-joins-and-set-ops.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-orderby-limit.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-orderby-limit.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-orderby-limit.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-orderby-limit.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-outside-filter.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-outside-filter.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-outside-filter.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-outside-filter.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-within-and-or.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-within-and-or.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/exists-subquery/exists-within-and-or.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/exists-subquery/exists-within-and-or.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-basic.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-basic.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-basic.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-basic.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-count-bug.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-count-bug.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-count-bug.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-group-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-group-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-group-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-having.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-having.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-having.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-having.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-joins.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-joins.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-joins.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-joins.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-limit.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-limit.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-limit.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-limit.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-multiple-columns.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-multiple-columns.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-multiple-columns.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-multiple-columns.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-null-semantics.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-null-semantics.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-null-semantics.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-null-semantics.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-nullability.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-nullability.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-nullability.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-nullability.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-order-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-order-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-order-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-order-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-set-operations.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-set-operations.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-set-operations.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-set-operations.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-subquery-in-join-condition.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-with-cte.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-with-cte.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/in-with-cte.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/in-with-cte.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/nested-not-in.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/nested-not-in.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/nested-not-in.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/nested-not-in.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-group-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-group-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-group-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-joins.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-joins.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column-literal.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-multi-column.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column-literal.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/not-in-unit-tests-single-column.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/simple-in.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/simple-in.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/in-subquery/simple-in.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/in-subquery/simple-in.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/invalid-correlation.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/invalid-correlation.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/invalid-correlation.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/invalid-correlation.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/subq-input-typecheck.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/subq-input-typecheck.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/negative-cases/subq-input-typecheck.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/negative-cases/subq-input-typecheck.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/resolution-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/resolution-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/resolution-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/resolution-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/nested-scalar-subquery-count-bug.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-count-bug.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-group-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-predicate.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-select.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-select.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-select.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-select.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/scalar-subquery/scalar-subquery-set-op.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-in-from.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/subquery-in-from.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-in-from.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/subquery-in-from.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-nested-data.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/subquery-nested-data.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-nested-data.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/subquery-nested-data.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-offset.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/subquery-offset.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/subquery/subquery-offset.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/subquery/subquery-offset.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/table-aliases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/table-aliases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/table-aliases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/table-aliases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/table-valued-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/table-valued-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/table-valued-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/table-valued-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/tablesample-negative.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/tablesample-negative.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/tablesample-negative.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/tablesample-negative.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/thetasketch.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/thetasketch.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/thetasketch.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/thetasketch.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/time.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/time.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/time.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/time.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestamp-ltz.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestamp-ltz.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestamp-ltz.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestamp-ltz.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestamp-ntz.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestamp-ntz.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestamp-ntz.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestamp-ntz.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestamp.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestamp.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestamp.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special-ansi.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special-ansi.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special-ansi.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special-ansi.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/datetime-special.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/datetime-special.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp-ansi.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp-ansi.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp-ansi.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp-ansi.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timestampNTZ/timestamp.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timestampNTZ/timestamp.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timezone.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timezone.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/timezone.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/timezone.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/to_from_avro.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/to_from_avro.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/to_from_avro.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/to_from_avro.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/transform.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/transform.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/transform.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/transform.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try-string-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try-string-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try-string-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try-string-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out.java21 b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out.java21
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_aggregates.sql.out.java21
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_aggregates.sql.out.java21
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_arithmetic.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_arithmetic.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_arithmetic.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_arithmetic.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_cast.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_cast.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_cast.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_cast.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_datetime_functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_datetime_functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_datetime_functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_datetime_functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_element_at.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_element_at.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_element_at.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_element_at.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_reflect.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_reflect.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/try_reflect.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/try_reflect.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/type-coercion-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/type-coercion-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/type-coercion-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/type-coercion-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/arrayJoin.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/arrayJoin.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/arrayJoin.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/arrayJoin.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/binaryComparison.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/binaryComparison.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/binaryComparison.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/binaryComparison.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/booleanEquality.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/booleanEquality.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/booleanEquality.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/booleanEquality.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/caseWhenCoercion.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/caseWhenCoercion.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/caseWhenCoercion.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/caseWhenCoercion.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/concat.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/concat.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/concat.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/concat.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/dateTimeOperations.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/dateTimeOperations.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/dateTimeOperations.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/dateTimeOperations.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/decimalPrecision.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/decimalPrecision.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/decimalPrecision.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/decimalPrecision.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/division.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/division.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/division.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/division.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/elt.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/elt.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/elt.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/elt.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/ifCoercion.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/ifCoercion.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/ifCoercion.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/ifCoercion.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/implicitTypeCasts.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/implicitTypeCasts.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/implicitTypeCasts.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/inConversion.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/inConversion.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/inConversion.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/inConversion.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapZipWith.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapZipWith.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapZipWith.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapZipWith.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapconcat.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapconcat.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/mapconcat.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/mapconcat.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/promoteStrings.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/promoteStrings.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/promoteStrings.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/promoteStrings.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/stringCastAndExpressions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/stringCastAndExpressions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/widenSetOperationTypes.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/widenSetOperationTypes.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/widenSetOperationTypes.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/widenSetOperationTypes.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/windowFrameCoercion.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/windowFrameCoercion.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/typeCoercion/native/windowFrameCoercion.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/typeCoercion/native/windowFrameCoercion.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-analytics.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-analytics.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-analytics.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-analytics.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by-ordinal.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by-ordinal.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by-ordinal.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by-ordinal.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-group-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-group-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-grouping-set.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-grouping-set.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf-grouping-set.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf-grouping-set.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udaf/udaf.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udaf/udaf.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part1.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part1.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part1.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part1.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part2.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part2.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part2.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part2.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part3.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part3.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part3.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part3.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part4.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part4.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-aggregates_part4.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-aggregates_part4.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-case.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-case.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-case.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-case.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_having.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_having.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_having.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_having.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_implicit.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_implicit.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/postgreSQL/udf-select_implicit.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/postgreSQL/udf-select_implicit.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-count.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-count.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-count.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-count.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-cross-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-cross-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-cross-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-cross-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-except-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-except-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-except-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-except-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-except.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-except.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-except.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-except.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-analytics.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-group-analytics.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-analytics.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-group-analytics.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-by.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-group-by.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-group-by.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-group-by.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-having.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-having.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-having.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-having.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-inline-table.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-inline-table.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-inline-table.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-inline-table.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-inner-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-inner-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-inner-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-inner-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-intersect-all.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-intersect-all.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-intersect-all.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-intersect-all.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-join-empty-relation.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-join-empty-relation.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-join-empty-relation.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-join-empty-relation.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-natural-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-natural-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-natural-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-natural-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-outer-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-outer-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-outer-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-outer-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-pivot.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-pivot.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-pivot.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-pivot.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-special-values.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-special-values.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-special-values.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-special-values.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-udaf.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-udaf.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-udaf.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-udaf.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-union.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-union.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-union.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-union.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-window.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-window.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udf/udf-window.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udf/udf-window.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udtf/udtf.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udtf/udtf.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/udtf/udtf.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/udtf/udtf.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/union-per-child-output-deduplication.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/union-per-child-output-deduplication.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/union-per-child-output-deduplication.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/union-per-child-output-deduplication.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/union-resolution-edge-cases.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/union-resolution-edge-cases.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/union-resolution-edge-cases.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/union-resolution-edge-cases.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/union.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/union.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/union.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/union.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/unpivot.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/unpivot.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/unpivot.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/unpivot.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/url-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/url-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/url-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/url-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/using-join.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/using-join.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/using-join.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/using-join.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/variant-field-extractions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/variant-field-extractions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/variant-field-extractions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/variant-field-extractions.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/variant/named-function-arguments.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/variant/named-function-arguments.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/variant/named-function-arguments.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/variant/named-function-arguments.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-conf.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-conf.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-conf.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-conf.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-binding-config.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-binding-config.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-binding-config.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-binding-config.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-binding.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-binding.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-binding.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-binding.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-compensation.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-compensation.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-compensation.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-compensation.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-evolution.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-evolution.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-evolution.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-evolution.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-type-evolution.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-type-evolution.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-schema-type-evolution.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-schema-type-evolution.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-with-default-collation.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-with-default-collation.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/view-with-default-collation.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/view-with-default-collation.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/window-with-aggregates.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/window-with-aggregates.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/window-with-aggregates.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/window-with-aggregates.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/window.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/window.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/window.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/window.sql.out
diff --git a/gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/xml-functions.sql.out b/gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/xml-functions.sql.out
similarity index 100%
rename from gluten-ut/spark41/src/test/resources/backends-velox/sql-tests/results/xml-functions.sql.out
rename to gluten-ut/spark41/src/test/resources/sql-tests-gluten/results/xml-functions.sql.out
diff --git a/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala b/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
index ea58cc7a5c..ce67445365 100644
--- a/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
+++ b/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala
@@ -19,9 +19,6 @@ package org.apache.gluten.utils.velox
import org.apache.gluten.utils.SQLQueryTestSettings
object VeloxSQLQueryTestSettings extends SQLQueryTestSettings {
- override def getResourceFilePath: String =
- getClass.getResource("/").getPath + "../../../src/test/resources/backends-velox/sql-tests"
-
override def getSupportedSQLQueryTests: Set[String] = SUPPORTED_SQL_QUERY_LIST
override def getOverwriteSQLQueryTests: Set[String] = OVERWRITE_SQL_QUERY_LIST
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 36/38: [GLUTEN][CORE] Share velox execution helpers in backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 94eb1d5628116d542c928f771d1f370570b21185
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Wed Jul 1 16:26:32 2026 +0800
[GLUTEN][CORE] Share velox execution helpers in backends-core
Move shared Arrow transition, collect-tail, and metrics updater logic into backends-core so velox can reuse common implementations and reduce duplicated backend glue.
---
.../SharedArrowColumnarToBackendColumnarExec.scala | 19 +++---
.../SharedHashAggregateMetricsUpdater.scala | 16 ++---
.../SharedInputIteratorMetricsUpdater.scala | 20 +++---
.../ArrowColumnarToVeloxColumnarExec.scala | 14 +---
.../metrics/HashAggregateMetricsUpdater.scala | 76 ++--------------------
.../metrics/InputIteratorMetricsUpdater.scala | 34 +++-------
6 files changed, 46 insertions(+), 133 deletions(-)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala b/backends-core/src/main/scala/org/apache/gluten/execution/SharedArrowColumnarToBackendColumnarExec.scala
similarity index 74%
copy from backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala
copy to backends-core/src/main/scala/org/apache/gluten/execution/SharedArrowColumnarToBackendColumnarExec.scala
index cca381ee84..0e77f15e07 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/execution/SharedArrowColumnarToBackendColumnarExec.scala
@@ -17,25 +17,28 @@
package org.apache.gluten.execution
import org.apache.gluten.backendsapi.arrow.ArrowBatchTypes.ArrowNativeBatchType
-import org.apache.gluten.backendsapi.velox.VeloxBatchType
-import org.apache.gluten.columnarbatch.VeloxColumnarBatches
import org.apache.gluten.extension.columnar.transition.Convention
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.vectorized.ColumnarBatch
-case class ArrowColumnarToVeloxColumnarExec(override val child: SparkPlan)
+/**
+ * Shared Arrow-native -> backend-native transition for backends whose only difference is the target
+ * batch type constant and the concrete `ColumnarBatch` conversion helper.
+ */
+abstract class SharedArrowColumnarToBackendColumnarExec(
+ child: SparkPlan,
+ backendBatchType: Convention.BatchType)
extends ColumnarToColumnarExec(child)
with GlutenColumnarToColumnarTransition {
+ protected def toBackendBatch(batch: ColumnarBatch): ColumnarBatch
+
override protected val from: Convention.BatchType = ArrowNativeBatchType
- override protected val to: Convention.BatchType = VeloxBatchType
+ override protected val to: Convention.BatchType = backendBatchType
override protected def mapIterator(in: Iterator[ColumnarBatch]): Iterator[ColumnarBatch] = {
- in.map(b => VeloxColumnarBatches.toVeloxBatch(b))
+ in.map(toBackendBatch)
}
-
- override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =
- copy(child = newChild)
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/SharedHashAggregateMetricsUpdater.scala
similarity index 88%
copy from backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala
copy to backends-core/src/main/scala/org/apache/gluten/metrics/SharedHashAggregateMetricsUpdater.scala
index dcfa0d7cf3..c87d815f24 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/metrics/SharedHashAggregateMetricsUpdater.scala
@@ -24,14 +24,14 @@ import org.apache.spark.task.TaskResources
import scala.collection.JavaConverters._
-trait HashAggregateMetricsUpdater extends MetricsUpdater {
+trait SharedHashAggregateMetricsUpdater extends MetricsUpdater {
def updateAggregationMetrics(
aggregationMetrics: java.util.ArrayList[OperatorMetrics],
aggParams: AggregationParams): Unit
}
-class HashAggregateMetricsUpdaterImpl(val metrics: Map[String, SQLMetric])
- extends HashAggregateMetricsUpdater {
+abstract class SharedHashAggregateMetricsUpdaterBase(val metrics: Map[String, SQLMetric])
+ extends SharedHashAggregateMetricsUpdater {
val aggOutputRows: SQLMetric = metrics("aggOutputRows")
val aggOutputVectors: SQLMetric = metrics("aggOutputVectors")
val aggOutputBytes: SQLMetric = metrics("aggOutputBytes")
@@ -44,7 +44,6 @@ class HashAggregateMetricsUpdaterImpl(val metrics: Map[String, SQLMetric])
val aggSpilledPartitions: SQLMetric = metrics("aggSpilledPartitions")
val aggSpilledFiles: SQLMetric = metrics("aggSpilledFiles")
val flushRowCount: SQLMetric = metrics("flushRowCount")
- val abandonedPartialAggregationRows: SQLMetric = metrics("abandonedPartialAggregationRows")
val loadedToValueHook: SQLMetric = metrics("loadedToValueHook")
val rowConstructionCpuCount: SQLMetric = metrics("rowConstructionCpuCount")
@@ -53,12 +52,9 @@ class HashAggregateMetricsUpdaterImpl(val metrics: Map[String, SQLMetric])
val extractionCpuCount: SQLMetric = metrics("extractionCpuCount")
val extractionWallNanos: SQLMetric = metrics("extractionWallNanos")
- val finalOutputRows: SQLMetric = metrics("finalOutputRows")
- val finalOutputVectors: SQLMetric = metrics("finalOutputVectors")
-
val loadLazyVectorTime: SQLMetric = metrics("loadLazyVectorTime")
- override def updateAggregationMetrics(
+ final override def updateAggregationMetrics(
aggregationMetrics: java.util.ArrayList[OperatorMetrics],
aggParams: AggregationParams): Unit = {
var idx = 0
@@ -82,8 +78,8 @@ class HashAggregateMetricsUpdaterImpl(val metrics: Map[String, SQLMetric])
aggSpilledPartitions += aggMetrics.spilledPartitions
aggSpilledFiles += aggMetrics.spilledFiles
flushRowCount += aggMetrics.flushRowCount
- abandonedPartialAggregationRows += aggMetrics.abandonedPartialAggregationRows
loadedToValueHook += aggMetrics.loadedToValueHook
+ updateAggSpecificMetrics(aggMetrics)
idx += 1
if (aggParams.rowConstructionNeeded) {
@@ -103,4 +99,6 @@ class HashAggregateMetricsUpdaterImpl(val metrics: Map[String, SQLMetric])
aggMetrics.spilledBytes)
}
}
+
+ protected def updateAggSpecificMetrics(aggMetrics: OperatorMetrics): Unit = {}
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala b/backends-core/src/main/scala/org/apache/gluten/metrics/SharedInputIteratorMetricsUpdater.scala
similarity index 73%
copy from backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala
copy to backends-core/src/main/scala/org/apache/gluten/metrics/SharedInputIteratorMetricsUpdater.scala
index ed006a5434..82ec8030a7 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala
+++ b/backends-core/src/main/scala/org/apache/gluten/metrics/SharedInputIteratorMetricsUpdater.scala
@@ -18,30 +18,30 @@ package org.apache.gluten.metrics
import org.apache.spark.sql.execution.metric.SQLMetric
-case class InputIteratorMetricsUpdater(metrics: Map[String, SQLMetric], forBroadcast: Boolean)
+abstract class SharedInputIteratorMetricsUpdater(
+ val metrics: Map[String, SQLMetric],
+ val forBroadcast: Boolean)
extends MetricsUpdater {
- override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
+
+ final override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
if (opMetrics != null) {
val operatorMetrics = opMetrics.asInstanceOf[OperatorMetrics]
metrics("cpuCount") += operatorMetrics.cpuCount
metrics("wallNanos") += operatorMetrics.wallNanos
if (!forBroadcast) {
if (operatorMetrics.outputRows == 0 && operatorMetrics.outputVectors == 0) {
- // Sometimes, velox does not update metrics for intermediate operator,
- // here we try to use the input metrics
+ // Sometimes, native backend does not update metrics for intermediate operator,
+ // here we try to use the input metrics.
metrics("numOutputRows") += operatorMetrics.inputRows
metrics("outputVectors") += operatorMetrics.inputVectors
} else {
metrics("numOutputRows") += operatorMetrics.outputRows
metrics("outputVectors") += operatorMetrics.outputVectors
}
- metrics.get("valueStreamDynamicFiltersAccepted").foreach {
- _ += operatorMetrics.numDynamicFiltersAccepted
- }
- metrics.get("valueStreamDynamicFilterInputRows").foreach {
- _ += operatorMetrics.numDynamicFilterInputRows
- }
+ updateExtraOutputMetrics(operatorMetrics)
}
}
}
+
+ protected def updateExtraOutputMetrics(operatorMetrics: OperatorMetrics): Unit = {}
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala b/backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala
index cca381ee84..bc205b5f3e 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/execution/ArrowColumnarToVeloxColumnarExec.scala
@@ -16,25 +16,17 @@
*/
package org.apache.gluten.execution
-import org.apache.gluten.backendsapi.arrow.ArrowBatchTypes.ArrowNativeBatchType
import org.apache.gluten.backendsapi.velox.VeloxBatchType
import org.apache.gluten.columnarbatch.VeloxColumnarBatches
-import org.apache.gluten.extension.columnar.transition.Convention
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.vectorized.ColumnarBatch
case class ArrowColumnarToVeloxColumnarExec(override val child: SparkPlan)
- extends ColumnarToColumnarExec(child)
- with GlutenColumnarToColumnarTransition {
+ extends SharedArrowColumnarToBackendColumnarExec(child, VeloxBatchType) {
- override protected val from: Convention.BatchType = ArrowNativeBatchType
-
- override protected val to: Convention.BatchType = VeloxBatchType
-
- override protected def mapIterator(in: Iterator[ColumnarBatch]): Iterator[ColumnarBatch] = {
- in.map(b => VeloxColumnarBatches.toVeloxBatch(b))
- }
+ override protected def toBackendBatch(batch: ColumnarBatch): ColumnarBatch =
+ VeloxColumnarBatches.toVeloxBatch(batch)
override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan =
copy(child = newChild)
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala b/backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala
index dcfa0d7cf3..2837a35d54 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/metrics/HashAggregateMetricsUpdater.scala
@@ -19,88 +19,22 @@ package org.apache.gluten.metrics
import org.apache.gluten.substrait.AggregationParams
import org.apache.spark.sql.execution.metric.SQLMetric
-import org.apache.spark.sql.utils.SparkMetricsUtil
-import org.apache.spark.task.TaskResources
-import scala.collection.JavaConverters._
-
-trait HashAggregateMetricsUpdater extends MetricsUpdater {
+trait HashAggregateMetricsUpdater extends SharedHashAggregateMetricsUpdater {
def updateAggregationMetrics(
aggregationMetrics: java.util.ArrayList[OperatorMetrics],
aggParams: AggregationParams): Unit
}
-class HashAggregateMetricsUpdaterImpl(val metrics: Map[String, SQLMetric])
- extends HashAggregateMetricsUpdater {
- val aggOutputRows: SQLMetric = metrics("aggOutputRows")
- val aggOutputVectors: SQLMetric = metrics("aggOutputVectors")
- val aggOutputBytes: SQLMetric = metrics("aggOutputBytes")
- val aggCpuCount: SQLMetric = metrics("aggCpuCount")
- val aggWallNanos: SQLMetric = metrics("aggWallNanos")
- val aggPeakMemoryBytes: SQLMetric = metrics("aggPeakMemoryBytes")
- val aggNumMemoryAllocations: SQLMetric = metrics("aggNumMemoryAllocations")
- val aggSpilledBytes: SQLMetric = metrics("aggSpilledBytes")
- val aggSpilledRows: SQLMetric = metrics("aggSpilledRows")
- val aggSpilledPartitions: SQLMetric = metrics("aggSpilledPartitions")
- val aggSpilledFiles: SQLMetric = metrics("aggSpilledFiles")
- val flushRowCount: SQLMetric = metrics("flushRowCount")
+class HashAggregateMetricsUpdaterImpl(override val metrics: Map[String, SQLMetric])
+ extends SharedHashAggregateMetricsUpdaterBase(metrics)
+ with HashAggregateMetricsUpdater {
val abandonedPartialAggregationRows: SQLMetric = metrics("abandonedPartialAggregationRows")
- val loadedToValueHook: SQLMetric = metrics("loadedToValueHook")
-
- val rowConstructionCpuCount: SQLMetric = metrics("rowConstructionCpuCount")
- val rowConstructionWallNanos: SQLMetric = metrics("rowConstructionWallNanos")
-
- val extractionCpuCount: SQLMetric = metrics("extractionCpuCount")
- val extractionWallNanos: SQLMetric = metrics("extractionWallNanos")
val finalOutputRows: SQLMetric = metrics("finalOutputRows")
val finalOutputVectors: SQLMetric = metrics("finalOutputVectors")
- val loadLazyVectorTime: SQLMetric = metrics("loadLazyVectorTime")
-
- override def updateAggregationMetrics(
- aggregationMetrics: java.util.ArrayList[OperatorMetrics],
- aggParams: AggregationParams): Unit = {
- var idx = 0
-
- if (aggParams.extractionNeeded) {
- extractionCpuCount += aggregationMetrics.get(idx).cpuCount
- extractionWallNanos += aggregationMetrics.get(idx).wallNanos
- idx += 1
- }
-
- val aggMetrics = aggregationMetrics.get(idx)
- aggOutputRows += aggMetrics.outputRows
- aggOutputVectors += aggMetrics.outputVectors
- aggOutputBytes += aggMetrics.outputBytes
- aggCpuCount += aggMetrics.cpuCount
- aggWallNanos += aggMetrics.wallNanos
- aggPeakMemoryBytes += aggMetrics.peakMemoryBytes
- aggNumMemoryAllocations += aggMetrics.numMemoryAllocations
- aggSpilledBytes += aggMetrics.spilledBytes
- aggSpilledRows += aggMetrics.spilledRows
- aggSpilledPartitions += aggMetrics.spilledPartitions
- aggSpilledFiles += aggMetrics.spilledFiles
- flushRowCount += aggMetrics.flushRowCount
+ override protected def updateAggSpecificMetrics(aggMetrics: OperatorMetrics): Unit = {
abandonedPartialAggregationRows += aggMetrics.abandonedPartialAggregationRows
- loadedToValueHook += aggMetrics.loadedToValueHook
- idx += 1
-
- if (aggParams.rowConstructionNeeded) {
- rowConstructionCpuCount += aggregationMetrics.get(idx).cpuCount
- rowConstructionWallNanos += aggregationMetrics.get(idx).wallNanos
- idx += 1
- }
-
- loadLazyVectorTime += aggregationMetrics.asScala.last.loadLazyVectorTime
-
- if (TaskResources.inSparkTask()) {
- SparkMetricsUtil.incMemoryBytesSpilled(
- TaskResources.getLocalTaskContext().taskMetrics(),
- aggMetrics.spilledInputBytes)
- SparkMetricsUtil.incDiskBytesSpilled(
- TaskResources.getLocalTaskContext().taskMetrics(),
- aggMetrics.spilledBytes)
- }
}
}
diff --git a/backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala b/backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala
index ed006a5434..f722a8f2a2 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/metrics/InputIteratorMetricsUpdater.scala
@@ -18,30 +18,16 @@ package org.apache.gluten.metrics
import org.apache.spark.sql.execution.metric.SQLMetric
-case class InputIteratorMetricsUpdater(metrics: Map[String, SQLMetric], forBroadcast: Boolean)
- extends MetricsUpdater {
- override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
- if (opMetrics != null) {
- val operatorMetrics = opMetrics.asInstanceOf[OperatorMetrics]
- metrics("cpuCount") += operatorMetrics.cpuCount
- metrics("wallNanos") += operatorMetrics.wallNanos
- if (!forBroadcast) {
- if (operatorMetrics.outputRows == 0 && operatorMetrics.outputVectors == 0) {
- // Sometimes, velox does not update metrics for intermediate operator,
- // here we try to use the input metrics
- metrics("numOutputRows") += operatorMetrics.inputRows
- metrics("outputVectors") += operatorMetrics.inputVectors
- } else {
- metrics("numOutputRows") += operatorMetrics.outputRows
- metrics("outputVectors") += operatorMetrics.outputVectors
- }
- metrics.get("valueStreamDynamicFiltersAccepted").foreach {
- _ += operatorMetrics.numDynamicFiltersAccepted
- }
- metrics.get("valueStreamDynamicFilterInputRows").foreach {
- _ += operatorMetrics.numDynamicFilterInputRows
- }
- }
+case class InputIteratorMetricsUpdater(
+ override val metrics: Map[String, SQLMetric],
+ override val forBroadcast: Boolean)
+ extends SharedInputIteratorMetricsUpdater(metrics, forBroadcast) {
+ override protected def updateExtraOutputMetrics(operatorMetrics: OperatorMetrics): Unit = {
+ metrics.get("valueStreamDynamicFiltersAccepted").foreach {
+ _ += operatorMetrics.numDynamicFiltersAccepted
+ }
+ metrics.get("valueStreamDynamicFilterInputRows").foreach {
+ _ += operatorMetrics.numDynamicFilterInputRows
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 28/38: [GLUTEN][CORE] Share OperatorMetrics and Metrics POJOs in backends-core
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit beea58c2e8a210b6e106ad6e0b1ea246f2deaea0
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Tue Jun 30 20:33:36 2026 +0800
[GLUTEN][CORE] Share OperatorMetrics and Metrics POJOs in backends-core
Lift the OperatorMetrics and Metrics JNI carrier POJOs out of
backends-velox into backends-core so both Velox- and Bolt-style native
backends can populate the same Java objects. The classes only depend on
IOperatorMetrics / IMetrics (gluten-substrait) and GlutenException
(gluten-core), which are already on the backends-core compile path.
No behaviour change: existing field set is preserved verbatim, only the
physical location moves. Backends keep loading these classes via the
same fully-qualified package name (`org.apache.gluten.metrics`).
---
.../src/main/java/org/apache/gluten/metrics/Metrics.java | 0
.../src/main/java/org/apache/gluten/metrics/OperatorMetrics.java | 0
2 files changed, 0 insertions(+), 0 deletions(-)
diff --git a/backends-velox/src/main/java/org/apache/gluten/metrics/Metrics.java b/backends-core/src/main/java/org/apache/gluten/metrics/Metrics.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/metrics/Metrics.java
rename to backends-core/src/main/java/org/apache/gluten/metrics/Metrics.java
diff --git a/backends-velox/src/main/java/org/apache/gluten/metrics/OperatorMetrics.java b/backends-core/src/main/java/org/apache/gluten/metrics/OperatorMetrics.java
similarity index 100%
rename from backends-velox/src/main/java/org/apache/gluten/metrics/OperatorMetrics.java
rename to backends-core/src/main/java/org/apache/gluten/metrics/OperatorMetrics.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 06/38: [GLUTEN][CORE] Expose ColumnarBatches.isLightBatch/ensureOffloaded as public
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 91483bcf4119c9f59848773e859531a802451609
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 11:10:57 2026 +0800
[GLUTEN][CORE] Expose ColumnarBatches.isLightBatch/ensureOffloaded as public
These two helpers are needed by out-of-package callers that produce or
consume native-based columnar batches (e.g. a custom parquet batch
writer under org.apache.spark.sql.execution.datasources), which cannot
reach the package-private methods. Promote them to public so backends
can reuse the batch type check and offload logic without duplicating it.
Generated-by: TraeCli openrouter-3o
---
.../main/java/org/apache/gluten/columnarbatch/ColumnarBatches.java | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/gluten-arrow/src/main/java/org/apache/gluten/columnarbatch/ColumnarBatches.java b/gluten-arrow/src/main/java/org/apache/gluten/columnarbatch/ColumnarBatches.java
index 40b931d92a..879d38f001 100644
--- a/gluten-arrow/src/main/java/org/apache/gluten/columnarbatch/ColumnarBatches.java
+++ b/gluten-arrow/src/main/java/org/apache/gluten/columnarbatch/ColumnarBatches.java
@@ -95,8 +95,7 @@ public final class ColumnarBatches {
* Light batch: Data is not readable from JVM, a long int handle (which is a pointer usually) is
* used to bind the batch to a native side implementation.
*/
- @VisibleForTesting
- static boolean isLightBatch(ColumnarBatch batch) {
+ public static boolean isLightBatch(ColumnarBatch batch) {
return identifyBatchType(batch) == BatchType.LIGHT;
}
@@ -128,7 +127,7 @@ public final class ColumnarBatches {
* Ensure the input batch is offloaded as native-based columnar batch (See {@link IndicatorVector}
* and {@link PlaceholderVector}). This method will close the input column batch after offloaded.
*/
- static ColumnarBatch ensureOffloaded(BufferAllocator allocator, ColumnarBatch batch) {
+ public static ColumnarBatch ensureOffloaded(BufferAllocator allocator, ColumnarBatch batch) {
if (ColumnarBatches.isLightBatch(batch)) {
return batch;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
(gluten) 07/38: [GLUTEN][CORE] Support HiveGenericUDTF in HiveUDFTransformer
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
taiyang-li pushed a commit to branch fake_add_bolt_backend
in repository https://gitbox.apache.org/repos/asf/gluten.git
commit 7bbd377877a2fb94f771d7e4be56e3e760c7d206
Author: liyang.127 <li...@bytedance.com>
AuthorDate: Sat Jun 27 11:52:50 2026 +0800
[GLUTEN][CORE] Support HiveGenericUDTF in HiveUDFTransformer
Recognize HiveGenericUDTF in HiveUDFTransformer.isHiveUDF and extract
its name/class in getHiveUDFNameAndClassName, so a Hive generic UDTF
can be mapped and offloaded to a native function (via UDFMappings)
like HiveSimpleUDF/HiveGenericUDF. No behavior change for backends
that do not register a UDTF mapping: unmapped UDTFs still throw
GlutenNotSupportException and fall back.
Generated-by: TraeCli openrouter-3o
---
.../src/main/scala/org/apache/spark/sql/hive/HiveUDFTransformer.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveUDFTransformer.scala b/gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveUDFTransformer.scala
index 4caf4fb43e..d30579fe2d 100644
--- a/gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveUDFTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveUDFTransformer.scala
@@ -26,7 +26,7 @@ import java.util.Locale
object HiveUDFTransformer {
def isHiveUDF(expr: Expression): Boolean = {
expr match {
- case _: HiveSimpleUDF | _: HiveGenericUDF => true
+ case _: HiveSimpleUDF | _: HiveGenericUDF | _: HiveGenericUDTF => true
case _ => false
}
}
@@ -36,6 +36,8 @@ object HiveUDFTransformer {
(s.name.stripPrefix("default."), s.funcWrapper.functionClassName)
case g: HiveGenericUDF =>
(g.name.stripPrefix("default."), g.funcWrapper.functionClassName)
+ case t: HiveGenericUDTF =>
+ (t.name.stripPrefix("default."), t.funcWrapper.functionClassName)
case _ =>
throw new GlutenNotSupportException(
s"Expression $expr is not a HiveSimpleUDF or HiveGenericUDF")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org