You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by we...@apache.org on 2026/03/16 08:23:46 UTC
(gluten-site) branch main updated: Add v1.5.0 archive documentation (#69)
This is an automated email from the ASF dual-hosted git repository.
weitingchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten-site.git
The following commit(s) were added to refs/heads/main by this push:
new 9a6f919 Add v1.5.0 archive documentation (#69)
9a6f919 is described below
commit 9a6f91974b5b4180cda8cf6eddaa977a323f93e8
Author: Wei-Ting Chen <we...@apache.org>
AuthorDate: Mon Mar 16 16:23:42 2026 +0800
Add v1.5.0 archive documentation (#69)
Add archived documentation for Apache Gluten (incubating) v1.5.0 release.
Docs are sourced from the apache/gluten repository at the v1.5.0 tag.
Co-authored-by: Copilot <22...@users.noreply.github.com>
---
_docs/v1.5.0/Configuration.md | 158 +++++
_docs/v1.5.0/developers/CppCodingStyle.md | 191 ++++++
_docs/v1.5.0/developers/HowTo.md | 211 ++++++
_docs/v1.5.0/developers/MicroBenchmarks.md | 480 ++++++++++++++
_docs/v1.5.0/developers/NewToGluten.md | 390 +++++++++++
_docs/v1.5.0/developers/PartialProject.md | 30 +
.../developers/ProfileMemoryOfGlutenWithVelox.md | 159 +++++
_docs/v1.5.0/developers/QueryTrace.md | 173 +++++
_docs/v1.5.0/developers/SubstraitModifications.md | 40 ++
_docs/v1.5.0/developers/UsingGperftoolsInCH.md | 33 +
_docs/v1.5.0/developers/UsingJemallocWithCH.md | 47 ++
.../v1.5.0/developers/VeloxDynamicSizingOffheap.md | 33 +
_docs/v1.5.0/developers/VeloxUDF.md | 246 +++++++
.../v1.5.0/developers/clickhouse-backend-debug.md | 97 +++
_docs/v1.5.0/developers/improvement-proposals.md | 100 +++
_docs/v1.5.0/developers/index.md | 11 +
_docs/v1.5.0/developers/velox-backend-CI.md | 42 ++
.../developers/velox-backend-build-in-docker.md | 68 ++
.../developers/velox-function-development-guide.md | 46 ++
_docs/v1.5.0/get-started/ClickHouse.md | 727 +++++++++++++++++++++
_docs/v1.5.0/get-started/GlutenUI.md | 37 ++
_docs/v1.5.0/get-started/Velox.md | 597 +++++++++++++++++
_docs/v1.5.0/get-started/VeloxABFS.md | 24 +
_docs/v1.5.0/get-started/VeloxGCS.md | 54 ++
_docs/v1.5.0/get-started/VeloxGPU.md | 81 +++
_docs/v1.5.0/get-started/VeloxIceberg.md | 120 ++++
_docs/v1.5.0/get-started/VeloxLocalCache.md | 20 +
_docs/v1.5.0/get-started/VeloxQAT.md | 155 +++++
_docs/v1.5.0/get-started/VeloxS3.md | 69 ++
_docs/v1.5.0/get-started/VeloxStageResourceAdj.md | 76 +++
_docs/v1.5.0/get-started/Work-with-pyspark.ipynb | 136 ++++
_docs/v1.5.0/get-started/build-guide.md | 76 +++
_docs/v1.5.0/get-started/index.md | 9 +
_docs/v1.5.0/index.md | 9 +
_docs/v1.5.0/velox-backend/index.md | 9 +
.../velox-backend-aggregate-function-support.md | 78 +++
.../velox-backend-generator-function-support.md | 23 +
.../velox-backend/velox-backend-limitations.md | 174 +++++
.../velox-backend-scalar-function-support.md | 458 +++++++++++++
.../velox-backend-support-progress.md | 125 ++++
.../velox-backend/velox-backend-troubleshooting.md | 99 +++
.../velox-backend-window-function-support.md | 25 +
42 files changed, 5736 insertions(+)
diff --git a/_docs/v1.5.0/Configuration.md b/_docs/v1.5.0/Configuration.md
new file mode 100644
index 0000000..f8b0ebb
--- /dev/null
+++ b/_docs/v1.5.0/Configuration.md
@@ -0,0 +1,158 @@
+---
+layout: page
+title: Configuration
+nav_order: 4
+parent: v1.5.0
+---
+<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO-GENERATED BY [org.apache.gluten.config.AllGlutenConfiguration] -->
+
+---
+
+layout: page
+title: Configuration
+nav_order: 15
+-------------
+
+## Spark base configurations for Gluten plugin
+
+| Key | Recommend Setting | Description |
+|-------------------------------|------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| spark.plugins | org.apache.gluten.GlutenPlugin | To load Gluten's components by Spark's plug-in loader. |
+| spark.memory.offHeap.enabled | true | Gluten use off-heap memory for certain operations. |
+| spark.memory.offHeap.size | 30G | The absolute amount of memory which can be used for off-heap allocation, in bytes unless otherwise specified.<br /> Note: Gluten Plugin will leverage this setting to allocate memory space for native usage even offHeap is disabled. <br /> The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Gluten Plugin. |
+| spark.shuffle.manager | org.apache.spark.shuffle.sort.ColumnarShuffleManager | To turn on Gluten Columnar Shuffle Plugin. |
+| spark.driver.extraClassPath | /path/to/gluten_jar_file | Gluten Plugin jar file to prepend to the classpath of the driver. |
+| spark.executor.extraClassPath | /path/to/gluten_jar_file | Gluten Plugin jar file to prepend to the classpath of executors. |
+
+## Gluten configurations
+
+| Key | Default | Description |
+|--------------------------------------------------------------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| spark.gluten.enabled | true | Whether to enable gluten. Default value is true. Just an experimental property. Recommend to enable/disable Gluten through the setting for spark.plugins. |
+| spark.gluten.execution.resource.expired.time | 86400 | Expired time of execution with resource relation has cached. |
+| spark.gluten.expression.blacklist | <undefined> | A black list of expression to skip transform, multiple values separated by commas. |
+| spark.gluten.loadLibFromJar | false | Whether to load shared libraries from jars. |
+| spark.gluten.loadLibOS | <undefined> | The shared library loader's OS name. |
+| spark.gluten.loadLibOSVersion | <undefined> | The shared library loader's OS version. |
+| spark.gluten.memory.isolation | false | Enable isolated memory mode. If true, Gluten controls the maximum off-heap memory can be used by each task to X, X = executor memory / max task slots. It's recommended to set true if Gluten serves concurrent queries within a single session, since not all memory Gluten allocated is guaranteed to be spillable. In the case, the feature should be enabled to avoid OOM. |
+| spark.gluten.memory.overAcquiredMemoryRatio | 0.3 | If larger than 0, Velox backend will try over-acquire this ratio of the total allocated memory as backup to avoid OOM. |
+| spark.gluten.memory.reservationBlockSize | 8MB | Block size of native reservation listener reserve memory from Spark. |
+| spark.gluten.numTaskSlotsPerExecutor | -1 | Must provide default value since non-execution operations (e.g. org.apache.spark.sql.Dataset#summary) doesn't propagate configurations using org.apache.spark.sql.execution.SQLExecution#withSQLConfPropagated |
+| spark.gluten.ras.costModel | legacy | The class name of user-defined cost model that will be used by Gluten's transition planner as well as by RAS. If not specified, a legacy built-in cost model will be used. The legacy cost model helps RAS planner exhaustively offload computations, and helps transition planner choose columnar-to-columnar transition over others. |
+| spark.gluten.ras.enabled | false | Enables RAS (relational algebra selector) during physical planning to generate more efficient query plan. Note, this feature doesn't bring performance profits by default. Try exploring option `spark.gluten.ras.costModel` for advanced usage. |
+| spark.gluten.shuffleWriter.bufferSize | <undefined> |
+| spark.gluten.soft-affinity.duplicateReading.maxCacheItems | 10000 | Enable Soft Affinity duplicate reading detection |
+| spark.gluten.soft-affinity.duplicateReadingDetect.enabled | false | If true, Enable Soft Affinity duplicate reading detection |
+| spark.gluten.soft-affinity.enabled | false | Whether to enable Soft Affinity scheduling. |
+| spark.gluten.soft-affinity.min.target-hosts | 1 | For on HDFS, if there are already target hosts, and then prefer to use the original target hosts to schedule |
+| spark.gluten.soft-affinity.replications.num | 2 | Calculate the number of the replications for scheduling to the target executors per file |
+| spark.gluten.sql.adaptive.costEvaluator.enabled | true | If true, use org.apache.spark.sql.execution.adaptive.GlutenCostEvaluator as custom cost evaluator class, else follow the configuration spark.sql.adaptive.customCostEvaluatorClass. |
+| spark.gluten.sql.ansiFallback.enabled | true | When true (default), Gluten will fall back to Spark when ANSI mode is enabled. When false, Gluten will attempt to execute in ANSI mode. |
+| spark.gluten.sql.broadcastNestedLoopJoinTransformerEnabled | true | Config to enable BroadcastNestedLoopJoinExecTransformer. |
+| spark.gluten.sql.cacheWholeStageTransformerContext | false | When true, `WholeStageTransformer` will cache the `WholeStageTransformerContext` when executing. It is used to get substrait plan node and native plan string. |
+| spark.gluten.sql.cartesianProductTransformerEnabled | true | Config to enable CartesianProductExecTransformer. |
+| spark.gluten.sql.collapseGetJsonObject.enabled | false | Collapse nested get_json_object functions as one for optimization. |
+| spark.gluten.sql.columnar.appendData | true | Enable or disable columnar v2 command append data. |
+| spark.gluten.sql.columnar.arrowUdf | true | Enable or disable columnar arrow udf. |
+| spark.gluten.sql.columnar.batchscan | true | Enable or disable columnar batchscan. |
+| spark.gluten.sql.columnar.broadcastExchange | true | Enable or disable columnar broadcastExchange. |
+| spark.gluten.sql.columnar.broadcastJoin | true | Enable or disable columnar broadcastJoin. |
+| spark.gluten.sql.columnar.cast.avg | true |
+| spark.gluten.sql.columnar.coalesce | true | Enable or disable columnar coalesce. |
+| spark.gluten.sql.columnar.collectLimit | true | Enable or disable columnar collectLimit. |
+| spark.gluten.sql.columnar.collectTail | true | Enable or disable columnar collectTail. |
+| spark.gluten.sql.columnar.enableNestedColumnPruningInHiveTableScan | true | Enable or disable nested column pruning in hivetablescan. |
+| spark.gluten.sql.columnar.enableVanillaVectorizedReaders | true | Enable or disable vanilla vectorized scan. |
+| spark.gluten.sql.columnar.executor.libpath || The gluten executor library path. |
+| spark.gluten.sql.columnar.expand | true | Enable or disable columnar expand. |
+| spark.gluten.sql.columnar.fallback.expressions.threshold | 50 | Fall back filter/project if number of nested expressions reaches this threshold, considering Spark codegen can bring better performance for such case. |
+| spark.gluten.sql.columnar.fallback.ignoreRowToColumnar | true | When true, the fallback policy ignores the RowToColumnar when counting fallback number. |
+| spark.gluten.sql.columnar.fallback.preferColumnar | true | When true, the fallback policy prefers to use Gluten plan rather than vanilla Spark plan if the both of them contains ColumnarToRow and the vanilla Spark plan ColumnarToRow number is not smaller than Gluten plan. |
+| spark.gluten.sql.columnar.filescan | true | Enable or disable columnar filescan. |
+| spark.gluten.sql.columnar.filter | true | Enable or disable columnar filter. |
+| spark.gluten.sql.columnar.force.hashagg | true | Whether to force to use gluten's hash agg for replacing vanilla spark's sort agg. |
+| spark.gluten.sql.columnar.forceShuffledHashJoin | true |
+| spark.gluten.sql.columnar.generate | true |
+| spark.gluten.sql.columnar.hashagg | true | Enable or disable columnar hashagg. |
+| spark.gluten.sql.columnar.hivetablescan | true | Enable or disable columnar hivetablescan. |
+| spark.gluten.sql.columnar.libname | gluten | The gluten library name. |
+| spark.gluten.sql.columnar.libpath || The gluten library path. |
+| spark.gluten.sql.columnar.limit | true |
+| spark.gluten.sql.columnar.maxBatchSize | 4096 |
+| spark.gluten.sql.columnar.parquet.write.blockSize | 128MB |
+| spark.gluten.sql.columnar.partial.project | true | Break up one project node into 2 phases when some of the expressions are non offload-able. Phase one is a regular offloaded project transformer that evaluates the offload-able expressions in native, phase two preserves the output from phase one and evaluates the remaining non-offload-able expressions using vanilla Spark projections |
+| spark.gluten.sql.columnar.physicalJoinOptimizationLevel | 12 | Fallback to row operators if there are several continuous joins. |
+| spark.gluten.sql.columnar.physicalJoinOptimizeEnable | false | Enable or disable columnar physicalJoinOptimize. |
+| spark.gluten.sql.columnar.preferStreamingAggregate | true | Velox backend supports `StreamingAggregate`. `StreamingAggregate` uses the less memory as it does not need to hold all groups in memory, so it could avoid spill. When true and the child output ordering satisfies the grouping key then Gluten will choose `StreamingAggregate` as the native operator. |
+| spark.gluten.sql.columnar.project | true | Enable or disable columnar project. |
+| spark.gluten.sql.columnar.project.collapse | true | Combines two columnar project operators into one and perform alias substitution |
+| spark.gluten.sql.columnar.query.fallback.threshold | -1 | The threshold for whether query will fall back by counting the number of ColumnarToRow & vanilla leaf node. |
+| spark.gluten.sql.columnar.range | true | Enable or disable columnar range. |
+| spark.gluten.sql.columnar.replaceData | true | Enable or disable columnar v2 command replace data. |
+| spark.gluten.sql.columnar.scanOnly | false | When enabled, only scan and the filter after scan will be offloaded to native. |
+| spark.gluten.sql.columnar.shuffle | true | Enable or disable columnar shuffle. |
+| spark.gluten.sql.columnar.shuffle.celeborn.fallback.enabled | true | If enabled, fall back to ColumnarShuffleManager when celeborn service is unavailable.Otherwise, throw an exception. |
+| spark.gluten.sql.columnar.shuffle.celeborn.useRssSort | true | If true, use RSS sort implementation for Celeborn sort-based shuffle.If false, use Gluten's row-based sort implementation. Only valid when `spark.celeborn.client.spark.shuffle.writer` is set to `sort`. |
+| spark.gluten.sql.columnar.shuffle.codec | <undefined> | By default, the supported codecs are lz4 and zstd. When spark.gluten.sql.columnar.shuffle.codecBackend=qat,the supported codecs are gzip and zstd. |
+| spark.gluten.sql.columnar.shuffle.codecBackend | <undefined> |
+| spark.gluten.sql.columnar.shuffle.compression.threshold | 100 | If number of rows in a batch falls below this threshold, will copy all buffers into one buffer to compress. |
+| spark.gluten.sql.columnar.shuffle.dictionary.enabled | false | Enable dictionary in hash-based shuffle. |
+| spark.gluten.sql.columnar.shuffle.merge.threshold | 0.25 |
+| spark.gluten.sql.columnar.shuffle.readerBufferSize | 1MB | Buffer size in bytes for shuffle reader reading input stream from local or remote. |
+| spark.gluten.sql.columnar.shuffle.realloc.threshold | 0.25 |
+| spark.gluten.sql.columnar.shuffle.sort.columns.threshold | 100000 | The threshold to determine whether to use sort-based columnar shuffle. Sort-based shuffle will be used if the number of columns is greater than this threshold. |
+| spark.gluten.sql.columnar.shuffle.sort.deserializerBufferSize | 1MB | Buffer size in bytes for sort-based shuffle reader deserializing raw input to columnar batch. |
+| spark.gluten.sql.columnar.shuffle.sort.partitions.threshold | 4000 | The threshold to determine whether to use sort-based columnar shuffle. Sort-based shuffle will be used if the number of partitions is greater than this threshold. |
+| spark.gluten.sql.columnar.shuffledHashJoin | true | Enable or disable columnar shuffledHashJoin. |
+| spark.gluten.sql.columnar.shuffledHashJoin.optimizeBuildSide | true | Whether to allow Gluten to choose an optimal build side for shuffled hash join. |
+| spark.gluten.sql.columnar.sort | true | Enable or disable columnar sort. |
+| spark.gluten.sql.columnar.sortMergeJoin | true | Enable or disable columnar sortMergeJoin. This should be set with preferSortMergeJoin=false. |
+| spark.gluten.sql.columnar.tableCache | false | Enable or disable columnar table cache. |
+| spark.gluten.sql.columnar.takeOrderedAndProject | true |
+| spark.gluten.sql.columnar.union | true | Enable or disable columnar union. |
+| spark.gluten.sql.columnar.wholeStage.fallback.threshold | -1 | The threshold for whether whole stage will fall back in AQE supported case by counting the number of ColumnarToRow & vanilla leaf node. |
+| spark.gluten.sql.columnar.window | true | Enable or disable columnar window. |
+| spark.gluten.sql.columnar.window.group.limit | true | Enable or disable columnar window group limit. |
+| spark.gluten.sql.columnarSampleEnabled | false | Disable or enable columnar sample. |
+| spark.gluten.sql.columnarToRowMemoryThreshold | 64MB |
+| spark.gluten.sql.countDistinctWithoutExpand | false | Convert Count Distinct to a UDAF called count_distinct to prevent SparkPlanner converting it to Expand+Count. WARNING: When enabled, count distinct queries will fail to fallback!!! |
+| spark.gluten.sql.extendedColumnPruning.enabled | true | Do extended nested column pruning for cases ignored by vanilla Spark. |
+| spark.gluten.sql.fallbackEncryptedParquet | false | If enabled, gluten will not offload scan when encrypted parquet files are detected |
+| spark.gluten.sql.fallbackEncryptedParquet.limit | 10 | If supplied, `limit` number of files will be checked to determine encryption and falling back java scan |
+| spark.gluten.sql.fallbackRegexpExpressions | false | If true, fall back all regexp expressions. There are a few incompatible cases between RE2 (used by native engine) and java.util.regex (used by Spark). User should enable this property if their incompatibility is intolerable. |
+| spark.gluten.sql.injectNativePlanStringToExplain | false | When true, Gluten will inject native plan tree to Spark's explain output. |
+| spark.gluten.sql.mergeTwoPhasesAggregate.enabled | true | Whether to merge two phases aggregate if there are no other operators between them. |
+| spark.gluten.sql.native.arrow.reader.enabled | false | This is config to specify whether to enable the native columnar csv reader |
+| spark.gluten.sql.native.bloomFilter | true |
+| spark.gluten.sql.native.hive.writer.enabled | true | This is config to specify whether to enable the native columnar writer for HiveFileFormat. Currently only supports HiveFileFormat with Parquet as the output file type. |
+| spark.gluten.sql.native.hyperLogLog.Aggregate | true |
+| spark.gluten.sql.native.parquet.write.blockRows | 100000000 |
+| spark.gluten.sql.native.union | false | Enable or disable native union where computation is completely offloaded to backend. |
+| spark.gluten.sql.native.writeColumnMetadataExclusionList | comment | Native write files does not support column metadata. Metadata in list would be removed to support native write files. Multiple values separated by commas. |
+| spark.gluten.sql.native.writer.enabled | <undefined> | This is config to specify whether to enable the native columnar parquet/orc writer |
+| spark.gluten.sql.orc.charType.scan.fallback.enabled | true | Force fallback for orc char type scan. |
+| spark.gluten.sql.removeNativeWriteFilesSortAndProject | true | When true, Gluten will remove the vanilla Spark V1Writes added sort and project for velox backend. |
+| spark.gluten.sql.rewrite.dateTimestampComparison | true | Rewrite the comparision between date and timestamp to timestamp comparison.For example `from_unixtime(ts) > date` will be rewritten to `ts > to_unixtime(date)` |
+| spark.gluten.sql.scan.fileSchemeValidation.enabled | true | When true, enable file path scheme validation for scan. Validation will fail if file scheme is not supported by registered file systems, which will cause scan operator fall back. |
+| spark.gluten.sql.supported.flattenNestedFunctions | and,or | Flatten nested functions as one for optimization. |
+| spark.gluten.sql.text.input.empty.as.default | false | treat empty fields in CSV input as default values. |
+| spark.gluten.sql.text.input.max.block.size | 8KB | the max block size for text input rows |
+| spark.gluten.sql.validation.printStackOnFailure | false |
+| spark.gluten.storage.hdfsViewfs.enabled | false | If enabled, gluten will convert the viewfs path to hdfs path in scala side |
+| spark.gluten.supported.hive.udfs || Supported hive udf names. |
+| spark.gluten.supported.python.udfs || Supported python udf names. |
+| spark.gluten.supported.scala.udfs || Supported scala udf names. |
+| spark.gluten.ui.enabled | true | Whether to enable the gluten web UI, If true, attach the gluten UI page to the Spark web UI. |
+
+## Gluten *experimental* configurations
+
+| Key | Default | Description [...]
+|-------------------------------------------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| spark.gluten.auto.adjustStageResource.enabled | false | Experimental: If enabled, gluten will try to set the stage resource according to stage execution plan. Only worked when aqe is enabled at the same time!! [...]
+| spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold | 0.5 | Experimental: Increase executor heap memory when stage contains fallen node count exceeds the total node count ratio. [...]
+| spark.gluten.auto.adjustStageResources.heap.ratio | 2.0 | Experimental: Increase executor heap memory when match adjust stage resource rule. [...]
+| spark.gluten.auto.adjustStageResources.offheap.ratio | 0.5 | Experimental: Decrease executor offheap memory when match adjust stage resource rule. [...]
+| spark.gluten.memory.dynamic.offHeap.sizing.enabled | false | Experimental: When set to true, the offheap config (spark.memory.offHeap.size) will be ignored and instead we will consider onheap and offheap memory in combination, both counting towards the executor memory config (spark.executor.memory). We will make use of JVM APIs to determine how much onheap memory is use, alongside tracking offheap allocations made by Gluten. We will then proceed to enforcing a total me [...]
+| spark.gluten.memory.dynamic.offHeap.sizing.memory.fraction | 0.6 | Experimental: Determines the memory fraction used to determine the total memory available for offheap and onheap allocations when the dynamic offheap sizing feature is enabled. The default is set to match spark.executor.memoryFraction. [...]
+| spark.gluten.sql.columnar.cudf | false | Enable or disable cudf support. This is an experimental feature. [...]
+
diff --git a/_docs/v1.5.0/developers/CppCodingStyle.md b/_docs/v1.5.0/developers/CppCodingStyle.md
new file mode 100644
index 0000000..96914ff
--- /dev/null
+++ b/_docs/v1.5.0/developers/CppCodingStyle.md
@@ -0,0 +1,191 @@
+---
+layout: page
+title: CppCodingStyle
+nav_order: 1
+parent: Developers
+grand_parent: v1.5.0
+---
+# Gluten CPP Core Guidelines
+
+This is a set of CPP core guidelines for Gluten. The aim is to make the codebase
+simpler, more efficient, more maintainable by promoting consistency and according
+to best practices.
+
+## Philosophy
+
+Philosophical rules are generally not measurable. However, they are valuable. For
+Gluten CPP coding, there are a few Philosophical rules as the following.
+
+* Write in ISO Standard C++.
+* Standard API first, the CPP programming APIs are priority to system calls.
+* Write code consistently. It's good for understanding and maintaining.
+* Keep simple, making code clear and easy to read.
+* Optimize code for reader, not for writer. Thus, more time will be spent reading
+ code than writing it.
+* Make it work, and then make it better or faster.
+* Don't import any complexity if possible. Collaborate with minimal knowledge
+ consensus.
+
+## Code Formatting
+
+Many aspects of C++ coding style will be covered by clang-format, such as spacing,
+line width, indentation and ordering (for includes, using directives and etc).
+
+* Always ensure your code is compatible with clang-format-15 for Velox backend.
+* `dev/formatcppcode.sh` is provided for formatting Velox CPP code.
+
+To format cmake files like CMakeLists.txt & *.cmake, `cmake-format` is required to
+be installed. Here is an example.
+
+```
+apt install python3-pip -y
+pip3 install --user cmake-format
+cmake-format --first-comment-is-literal True --in-place cpp/velox/CMakeLists.txt
+```
+
+## Naming Conventions
+
+* Use **PascalCase** for types (class, struct, enum, type alias, type
+ template parameter) and file name.
+* Use **camelCase** for function, member and local variable, and non-type
+ template parameter.
+* Use **camelCase_** for private and protected member variable.
+* Use **snake_case** for namespace name and build target.
+* Use **UPPER_SNAKE_CASE** for macro.
+* Use **kPascalCase** for static constant and enumerator.
+
+## Designs
+
+* No over design.
+* No negation of negation, `isValid` is better than `isNotInvalid`.
+* Avoid corner case, and common case first.
+* Express ideas directly, don't let me think.
+* Make a check for the arguments in the interface between modules, and don't make a check
+ in the inner implementation, use `assert` in the private implementation instead of too
+ much safe check.
+
+## Source File & Header File
+
+* All header files must have a single-inclusion guard using `#pragma once`
+* Always use `.h` as header file suffix, not `.hpp`.
+* Always use `.cc` as source file suffix, neither `.cpp` nor `.cxx`.
+* One file should contain one main class, and the file name should be consistent with
+ the main class name.
+ * Obvious exception: files used for defining various misc functions.
+* If a header file has a corresponding source file, they should have the same file
+ name with different suffix, such as `a.h vs a.cc`.
+* If a function is declared in the file `a.h`, ensure it's defined in the corresponding
+ source file `a.cc`, do not define it in other files.
+* No deep source directory for CPP files, not do it as JAVA.
+* Include header files should satisfy the following rules.
+ * Include the necessary header files, which means the source file (.cc) containing
+ the only one line `#include "test.h"` can be compiled successfully without
+ including any other header files.
+ * Do not include any unnecessary header files, the more including, the slower
+ compiling.
+ * In one word, no more, no less, just as needed.
+
+## Class
+
+* Base class name doesn't end with Base, use `Backend` instead of `BackendBase`.
+* Ensure one class does one thing, and follows the single responsibility principle.
+* No big class, No huge class, No too much interfaces.
+* Distinguish interface from implementation, make implementations private.
+* When designing a class hierarchy, distinguish between interface inheritance
+ and implementation inheritance.
+ * Ensure that public inheritance represent the relation of `is-a`.
+ * Ensure that private inheritance represent the relation of `implements-with`.
+* Don't make a function `virtual` without reason.
+* Ensure the polymorphic base class has a `virtual` destructor.
+* Use `override` to make overriding explicit and to make the compiler work.
+* Use `const` to mark the member function read-only as far as possible.
+* When you try to define a `copy constructor` or a `operator=` for a class, remember
+ the `Rule of three/five/zero`.
+
+## Function
+
+* Make functions short and simple.
+* Calling a meaningful function is more readable than writing too many statements
+ in place, but the performance-sensitive code path is an exception.
+* Give the function a good name, how to check whether the function name is good
+ or not.
+ * When you read it loudly, you feel smooth.
+ * The information can be represented by arguments should not be encoded into
+ the function name. such as. use `get(size_t index)` instead of `getByIndex`.
+* A function should focus on a single logic operation.
+* A function should do as the name meaning.
+ * do everything covered by the function name
+ * don't do anything not covered by the function name
+
+## Variable
+
+* Make variable names simple and meaningful.
+* Don't group all your variables at the top of the scope, it's an outdated habit.
+* Declare variables as close to the usage point as possible.
+
+## Constant
+
+* Prefer const variables to using preprocessor (`#define`) to define constant values.
+
+## Macro
+
+* Macros downgrade readability, break mind, and affect debug.
+* Macros have side effects.
+* Use macros cautiously and carefully.
+* Consider using `const` variables or `inline` functions to replace macros.
+* Consider defining macros with the wrap of `do {...} while (0)`
+* Avoid using 3rd party library macros directly.
+
+## Namespace
+
+* Don't `using namespace xxx` in header files. Instead, you can do this in source files.
+ But it's still not encouraged.
+* Place all Gluten CPP codes under `namespace gluten` because one level namespace
+ is enough. No nested namespace. Nested namespaces bring mess.
+* The anonymous namespace is recommended for defining file level classes, functions
+ and variables. It's used to place file scoped static functions and variables.
+
+## Resource Management
+
+* Use handles and RAII to manage resources automatically.
+* Immediately give the result of an explicit resource allocation to a manager
+ object.
+* Prefer scoped objects and stack objects.
+* Use raw pointers to denote individual objects.
+* Use `pointer + size_t` to denote array objects if you don't want to use containers.
+* A raw pointer (a `T*`) is non-owning.
+* A raw reference (a `T&`) is non-owning.
+* Understand the difference of `unique_ptr`, `shared_ptr`, `weak_ptr`.
+ * `unique_ptr` represents ownership, but not share ownership. `unique_ptr` is
+ equivalent to RAII, release the resource when the object is destructed.
+ * `shared_ptr` represents shared ownership by use-count. It is more expensive
+ that `unique_ptr`.
+ * `weak_ptr` models temporary ownership. It is useful in breaking reference cycles
+ formed by objects managed by `shared_ptr`.
+* Use `unique_ptr` or `shared_ptr` to represent ownership.
+* Prefer `unique_ptr` over `shared_ptr` unless you need to share ownership.
+* Use `make_unique` to make `unique_ptr`s.
+* Use `make_shared` to make `shared_ptr`s.
+* Take smart pointers as parameters only to explicitly express lifetime semantics.
+* **For general use**, take `T*` or `T&` arguments rather than smart pointers.
+
+## Exception
+
+* The exception specifications are changing always. The difference between various CPP
+ standards is big, so we should use exception cautiously in Gluten.
+* Prefer `return code` to throwing exceptions.
+* Prefer compile-time checking to run-time checking.
+* Encapsulate messy constructors, rather than spreading through the code.
+
+## Code Comment
+
+* Add necessary comments. The comment is not the more the better, also not the
+ less the better.
+* Good comment makes obscure code easily understood. It's unnecessary to add comments for
+ quite obvious code.
+
+## References
+
+* [CppCoreGuidelines](https://github.com/fluz/CppCoreGuidelines)
+* [Velox CODING_STYLE](https://github.com/facebookincubator/velox/blob/main/CODING_STYLE.md)
+* Thanks Gluten developers for their wise suggestions and helps.
diff --git a/_docs/v1.5.0/developers/HowTo.md b/_docs/v1.5.0/developers/HowTo.md
new file mode 100644
index 0000000..623a016
--- /dev/null
+++ b/_docs/v1.5.0/developers/HowTo.md
@@ -0,0 +1,211 @@
+---
+layout: page
+title: HowTo
+nav_order: 2
+parent: Developers
+grand_parent: v1.5.0
+---
+There are some common questions about developing, debugging and testing been asked again and again. In order to help the developers to contribute
+to Gluten as soon as possible, we collected these frequently asked questions, and organized them in the form of Q&A. It's convenient for the developers
+to check and learn.
+
+When you encountered a new problem and then resolved it, please add a new item to this document if you think it may be helpful to the other developers.
+
+We use `${GLUTEN_HOME}` to represent the home directory of Gluten in this document.
+
+# How to understand the key work of Gluten?
+
+The Gluten worked as the role of bridge, it's a middle layer between the Spark and the native execution library.
+
+The Gluten is responsible for validating whether the operators of the Spark plan can be executed by the native engine or not. If yes, the Gluten
+transforms Spark plan to Substrait plan, and then send the Substrait plan to the native engine.
+
+The Gluten codes consist of two parts: the C++ codes and the Java/Scala codes.
+1. All C++ codes are placed under the directory of `${GLUTEN_HOME}/cpp`, the Java/Scala codes are placed under several directories, such as
+ `${GLUTEN_HOME}/gluten-substrait` `${GLUTEN_HOME}/gluten-data` `${GLUTEN_HOME}/backends-velox`.
+2. The Java/Scala codes are responsible for validating and transforming the execution plan. Source data should also be provided, the source data may
+ come from files or other forms such as networks.
+3. The C++ codes take the Substrait plan and the source data as inputs and transform the Substrait plan to the corresponding backend plan. If the backend
+ is Velox, the Substrait plan will be transformed to the Velox plan, and then be executed.
+
+JNI is a programming technology of invoking C++ from Java. All JNI interfaces are defined in the file `JniWrapper.cc` under the directory `jni`.
+
+# How to debug in Gluten?
+
+## 1 How to debug C++
+If you don't concern about the Scala/Java codes and just want to debug the C++ codes executed in native engine, you may debug the C++ via benchmarks
+with GDB.
+
+To debug C++, you have to generate the example files, the example files consist of:
+- A file contained Substrait plan in JSON format
+- One or more input data files in Parquet format
+
+You can generate the example files by the following steps:
+
+1. Build Velox and Gluten CPP:
+
+```
+${GLUTEN_HOME}/dev/builddeps-veloxbe.sh --build_tests=ON --build_benchmarks=ON --build_examples=ON --build_type=Debug
+```
+
+- Compiling with `--build_type=Debug` is good for debugging.
+- The executable file `generic_benchmark` will be generated under the directory of `gluten_home/cpp/build/velox/benchmarks/`.
+
+2. Build Gluten and generate the example files:
+
+```
+cd ${GLUTEN_HOME}
+mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox \
+-am -DtagsToInclude="org.apache.gluten.tags.GenerateExample" \
+-Dtest=none -DfailIfNoTests=false \
+-Dexec.skip
+```
+
+- After the above operations, the example files are generated under `${GLUTEN_HOME}/backends-velox`
+- You can check it by the command `tree ${GLUTEN_HOME}/backends-velox/generated-native-benchmark/`
+- You may replace `-Pspark-3.2` with `-Pspark-3.3` if your spark's version is 3.3
+
+```shell
+$ tree ${GLUTEN_HOME}/backends-velox/generated-native-benchmark/
+/some-dir-to-gluten-home/backends-velox/generated-native-benchmark/
+|-- conf_12_0.ini
+|-- data_12_0_0.parquet
+|-- data_12_0_1.parquet
+`-- plan_12_0.json
+```
+
+3. Now, run benchmarks with GDB
+
+```shell
+cd ${GLUTEN_HOME}
+gdb cpp/build/velox/benchmarks/generic_benchmark
+```
+
+- When GDB load `generic_benchmark` successfully, you can set `breakpoint` on the `main` function with command `b main`, and then run using the `r` command with
+ arguments for the example files like:
+ ```
+ r --with-shuffle --partitioning hash --threads 1 --iterations 1 \
+ --conf backends-velox/generated-native-benchmark/conf_12_0.ini \
+ --plan backends-velox/generated-native-benchmark/plan_12_0.json \
+ --data backends-velox/generated-native-benchmark/data_12_0_0.parquet,backends-velox/generated-native-benchmark/data_12_0_1.parquet
+ ```
+ The process `generic_benchmark` will start and stop at the `main` function.
+- You can check the variables' state with command `p variable_name`, or execute the program line by line with command `n`, or step-in the function been
+ called with command `s`.
+- Actually, you can debug `generic_benchmark` with any gdb commands as debugging normal C++ program, because the `generic_benchmark` is a pure C++
+ executable file in fact.
+
+4. `gdb-tui` is a valuable feature and is worth trying. You can get more help from the online docs.
+[gdb-tui](https://sourceware.org/gdb/onlinedocs/gdb/TUI.html)
+
+5. You can start `generic_benchmark` with specific JSON plan and input files
+- You can also edit the file `plan_12_0.json` to custom the Substrait plan or specify the inputs files placed in the other directory.
+
+6. Get more detail information about benchmarks from [MicroBenchmarks](./MicroBenchmarks.md)
+
+## 2 How to debug plan validation process
+
+Gluten will validate generated plan before execute it, and validation usually happens in native side, so we provide a utility to help debug validation process in native side.
+
+1. Run query with conf `spark.gluten.sql.debug=true`, and you will find generated plan be printed in stderr with json format, save it as `plan.json` for example.
+2. Compile cpp part with `--build_benchmarks=ON`, then check `plan_validator_util` executable file in `${GLUTEN_HOME}/cpp/build/velox/benchmarks/`.
+3. Run or debug with `./plan_validator_util <path>/plan.json`
+
+## 3 How to debug Java/Scala
+
+To debug some runtime issues in Scala/Java, we recommend developers to use Intellij remote debug, see [tutorial link](https://www.jetbrains.com/help/idea/tutorial-remote-debug.html).
+
+According to your setting for Intellij remote debug, please set `SPARK_SUBMIT_OPTS` in the environment where spark-submit is executed. See the below example.
+
+```
+export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8008
+```
+
+To run a Scala/Java test class, you can use the below mvn command (take Velox backend as example), which is helpful to debug some unit test failure reported by Gluten CI.
+```
+mvn test -Pspark-3.5 -Pspark-ut -Pbackends-velox -DargLine="-Dspark.test.home=/path/to/spark/source/code/home/" -DwildcardSuites=xxx
+```
+
+Please set `wildcardSuites` with a fully qualified class name. `spark.test.home` is optional to set. It is only required for some test suites to use Spark resources.
+If you are specifying the `spark.test.home` arg, it should be set to either:
+* The path a directory containing Spark source code, which has already been built
+* Or use the `install_spark_resources.sh` script to get a directory with the necessary resource files:
+ ```
+ # Define a directory to use for the Spark files and the Spark version
+ export spark_dir=/tmp/spark
+ export spark_version=3.5
+
+ # Run the install_spark_resources.sh script
+ .github/workflows/util/install_spark_resources.sh ${spark_version} ${spark_dir}
+ ```
+ After running the `install_spark_resources.sh`, define the `spark.test.home` directory like:
+ `-DargLine="-Dspark.test.home=${spark_dir}/shims/spark35/spark_home"` when running unit tests.
+
+For most cases, please make sure Gluten native build is done before running a Scala/Java test.
+
+## 4 How to debug with core-dump
+wait to complete
+
+```shell
+cd the_directory_of_core_file_generated
+gdb ${GLUTEN_HOME}/cpp/build/releases/libgluten.so 'core-Executor task l-2000883-1671542526'
+
+```
+- the `core-Executor task l-2000883-1671542526` represents the core file name.
+
+# How to use jemalloc for Gluten native engine
+
+Currently, we have no dedicated memory allocator implemented by jemalloc. User can set environment variable `LD_PRELOAD` for lib jemalloc
+to let it override the corresponding C standard functions entirely. It may help alleviate OOM issues.
+
+`spark.executorEnv.LD_PREALOD=/path/to/libjemalloc.so`
+
+# How to run TPC-H on Velox backend
+
+Now, both Parquet and DWRF format files are supported, related scripts and files are under the directory of `${GLUTEN_HOME}/backends-velox/workload/tpch`.
+The file `README.md` under `${GLUTEN_HOME}/backends-velox/workload/tpch` offers some useful help, but it's still not enough and exact.
+
+One way of run TPC-H test is to run velox-be by workflow, you can refer to [velox_backend.yml](https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_backend.yml#L280)
+
+Here we will explain how to run TPC-H on Velox backend with the Parquet file format.
+1. First, prepare the datasets, you have two choices.
+ - One way, generate Parquet datasets using the script under `${GLUTEN_HOME}/tools/workload/tpch/gen_data/parquet_dataset`, you can get help from the above
+ -mentioned `README.md`.
+ - The other way, using the small dataset under `${GLUTEN_HOME}/backends-velox/src/test/resources/tpch-data-parquet` directly, if you just want to make simple
+ TPC-H testing, this dataset is a good choice.
+2. Second, run TPC-H on Velox backend testing.
+ - Modify `${GLUTEN_HOME}/tools/workload/tpch/run_tpch/tpch_parquet.scala`.
+ - Set `var parquet_file_path` to correct directory. If using the small dataset directly in the step one, then modify it as below:
+
+ ```scala
+ var parquet_file_path = "gluten_home/backends-velox/src/test/resources/tpch-data-parquet"
+ ```
+
+ - Set `var gluten_root` to correct directory. If `${GLUTEN_HOME}` is the directory of `/home/gluten`, then modify it as below
+
+ ```scala
+ var gluten_root = "/home/gluten"
+ ```
+
+ - Modify `${GLUTEN_HOME}/tools/workload/tpch/run_tpch/tpch_parquet.sh`.
+ - Set `GLUTEN_JAR` correctly. Please refer to the section of [Build Gluten with Velox Backend](../get-started/Velox.md#build-gluten-with-velox-backend)
+ - Set `SPARK_HOME` correctly.
+ - Set the memory configurations appropriately.
+ - Execute `tpch_parquet.sh` using the below command.
+ - `cd ${GLUTEN_HOME}/tools/workload/tpch/run_tpch/`
+ - `./tpch_parquet.sh`
+
+# How to run TPC-DS
+
+wait to add
+
+# How to track the memory exhaust problem
+
+When your gluten spark jobs failed because of OOM, you can track the memory allocation's call stack by configuring `spark.gluten.memory.backtrace.allocation = true`.
+The above configuration will use `BacktraceAllocationListener` wrapping from `SparkAllocationListener` to create `VeloxMemoryManager`.
+
+`BacktraceAllocationListener` will check every allocation, if a single allocation bytes exceeds a fixed value or the accumulative allocation bytes exceeds 1/2/3...G,
+the call stack of memory allocation will be outputted to standard output, you can check the backtrace and get some valuable information about tracking the memory exhaust issues.
+
+You can also adjust the policy to decide when to backtrace, such as the fixed value.
+
diff --git a/_docs/v1.5.0/developers/MicroBenchmarks.md b/_docs/v1.5.0/developers/MicroBenchmarks.md
new file mode 100644
index 0000000..8d875f6
--- /dev/null
+++ b/_docs/v1.5.0/developers/MicroBenchmarks.md
@@ -0,0 +1,480 @@
+---
+layout: page
+title: MicroBenchmarks
+nav_order: 3
+parent: Developers
+grand_parent: v1.5.0
+---
+# Generate Micro Benchmarks for Velox Backend
+
+**This document explains how to use the existing micro benchmark template in Gluten Cpp.**
+
+A micro benchmark for Velox backend is provided in Gluten Cpp to simulate the execution of a first
+or middle stage in Spark. It serves as a more convenient alternative to debug in Gluten Cpp
+comparing with directly debugging in a Spark job. Developers can use it to create their own
+workloads, debug in native process, profile the hotspot and do optimizations.
+
+To simulate a first stage, you need to dump the Substrait plan and input split info into two JSON
+files. The input URIs of the splits should be existing file locations, which can be either local or
+HDFS paths.
+
+To simulate a middle stage, in addition to the JSON file, you also need to save the input data of
+this stage into Parquet files. The benchmark will load the data into Arrow format, then add
+Arrow2Velox to feed the data into Velox pipeline to reproduce the reducer stage. Shuffle exchange is
+not included.
+
+Please refer to the sections below to learn how to dump the Substrait plan and create the input data
+files.
+
+## Try the example
+
+To run a micro benchmark, user should provide one file that contains the Substrait plan in JSON
+format, and optional one or more input data files in parquet format. The commands below help to
+generate example input files:
+
+```shell
+cd /path/to/gluten/
+./dev/buildbundle-veloxbe.sh --build_tests=ON --build_benchmarks=ON
+
+# Run test to generate input data files. If you are using spark 3.3, replace -Pspark-3.2 with -Pspark-3.3.
+mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox -am \
+-DtagsToInclude="org.apache.gluten.tags.GenerateExample" -Dtest=none -DfailIfNoTests=false -Dexec.skip
+```
+
+The generated example files are placed in gluten/backends-velox:
+
+- stageId means Spark stage id in web ui.
+- partitionedId means Spark partition id in web stage ui.
+- vId means backends internal virtual id.
+
+```shell
+$ tree gluten/backends-velox/generated-native-benchmark/
+gluten/backends-velox/generated-native-benchmark/
+├── conf_12_10_3.ini
+├── data_12_10_3_0.parquet
+├── data_12_10_3_1.parquet
+├── plan_12_10_3.json
+```
+
+Run micro benchmark with the generated files as input. You need to specify the **absolute** path to
+the input files:
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan <path-to-gluten>/backends-velox/generated-native-benchmark/plan_{stageId}_{partitionId}_{vId}.json \
+--data <path-to-gluten>/backends-velox/generated-native-benchmark/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet,\
+<path-to-gluten>/backends-velox/generated-native-benchmark/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet \
+--conf <path-to-gluten>/backends-velox/generated-native-benchmark/conf_{stageId}_{partitionId}_{vId}.ini \
+--threads 1 --iterations 1 --noprint-result
+```
+
+The output should be like:
+
+```shell
+2022-11-18T16:49:56+08:00
+Running ./generic_benchmark
+Run on (192 X 3800 MHz CPU s)
+CPU Caches:
+ L1 Data 48 KiB (x96)
+ L1 Instruction 32 KiB (x96)
+ L2 Unified 2048 KiB (x96)
+ L3 Unified 99840 KiB (x2)
+Load Average: 0.28, 1.17, 1.59
+***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
+-- Project[expressions: (n3_0:BIGINT, ROW["n1_0"]), (n3_1:VARCHAR, ROW["n1_1"])] -> n3_0:BIGINT, n3_1:VARCHAR
+ Output: 535 rows (65.81KB, 1 batches), Cpu time: 36.33us, Blocked wall time: 0ns, Peak memory: 1.00MB, Memory allocations: 3, Threads: 1
+ queuedWallNanos sum: 2.00us, count: 2, min: 0ns, max: 2.00us
+ -- HashJoin[RIGHT SEMI (FILTER) n0_0=n1_0] -> n1_0:BIGINT, n1_1:VARCHAR
+ Output: 535 rows (65.81KB, 1 batches), Cpu time: 191.56us, Blocked wall time: 0ns, Peak memory: 2.00MB, Memory allocations: 8
+ HashBuild: Input: 582 rows (16.45KB, 1 batches), Output: 0 rows (0B, 0 batches), Cpu time: 1.84us, Blocked wall time: 0ns, Peak memory: 1.00MB, Memory allocations: 3, Threads: 1
+ distinctKey0 sum: 583, count: 1, min: 583, max: 583
+ queuedWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ rangeKey0 sum: 59748, count: 1, min: 59748, max: 59748
+ HashProbe: Input: 37897 rows (296.07KB, 1 batches), Output: 535 rows (65.81KB, 1 batches), Cpu time: 189.71us, Blocked wall time: 0ns, Peak memory: 1.00MB, Memory allocations: 5, Threads: 1
+ queuedWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ -- ArrowStream[] -> n0_0:BIGINT
+ Input: 0 rows (0B, 0 batches), Output: 37897 rows (296.07KB, 1 batches), Cpu time: 1.29ms, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+ -- ArrowStream[] -> n1_0:BIGINT, n1_1:VARCHAR
+ Input: 0 rows (0B, 0 batches), Output: 582 rows (16.45KB, 1 batches), Cpu time: 894.22us, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+
+-----------------------------------------------------------------------------------------------------------------------------
+Benchmark Time CPU Iterations UserCounters...
+-----------------------------------------------------------------------------------------------------------------------------
+InputFromBatchVector/iterations:1/process_time/real_time/threads:1 41304520 ns 23740340 ns 1 collect_batch_time=34.7812M elapsed_time=41.3113M
+
+```
+
+## Generate Substrait plan and input for any query
+
+First, build Gluten with `--build_benchmarks=ON`.
+
+```shell
+cd /path/to/gluten/
+./dev/buildbundle-veloxbe.sh --build_benchmarks=ON
+
+# For debugging purpose, rebuild Gluten with build type `Debug`.
+./dev/buildbundle-veloxbe.sh --build_benchmarks=ON --build_type=Debug
+```
+
+First, get the Stage Id from spark UI for the stage you want to simulate.
+And then re-run the query with below configurations to dump the inputs to micro benchmark.
+
+| Parameters | Description | Recommend Setting |
+|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|
+| spark.gluten.sql.benchmark_task.taskId | Comma-separated string to specify the Task IDs to dump. If it's set, `spark.gluten.sql.benchmark_task.stageId` and `spark.gluten.sql.benchmark_task.partitionId` will be ignored. | Comma-separated string of task IDs. Empty by default. |
+| spark.gluten.sql.benchmark_task.stageId | Spark stage ID. | Target stage ID |
+| spark.gluten.sql.benchmark_task.partitionId | Comma-separated string to specify the Partition IDs in a stage to dump. Must be specified together with `spark.gluten.sql.benchmark_task.stageId`. Empty by default, meaning all partitions of this stage will be dumped. To identify the partition ID, navigate to the `Stage` tab in the Spark UI and locate it under the `Index` column. | Comma-separated string of partition IDs. Empty by default. |
+| spark.gluten.saveDir | Directory to save the inputs to micro benchmark, should exist and be empty. | /path/to/saveDir |
+
+Check the files in `spark.gluten.saveDir`. If the simulated stage is a first stage, you will get 3
+or 4 types of dumped file:
+
+- Configuration file: INI formatted, file name `conf_{stageId}_{partitionId}_{vId}.ini`. Contains the
+ configurations to init Velox backend and runtime session.
+- Plan file: JSON formatted, file name `plan_{stageId}_{partitionId}_{vId}.json`. Contains the substrait
+ plan to the stage, without input file splits.
+- Split file: JSON formatted, file name `split_{stageId}_{partitionId}_{vId}_{splitIdx}.json`. There can
+ be more than one split file in a first stage task. Contains the substrait plan piece to the input
+ file splits.
+- Data file(optional): Parquet formatted, file
+ name `data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet`. If the first stage contains one or
+ more BHJ operators, there can be one or more input data files. The input data files of a first
+ stage will be loaded as iterators to serve as the inputs for the pipeline:
+
+```
+"localFiles": {
+ "items": [
+ {
+ "uriFile": "iterator:0"
+ }
+ ]
+}
+```
+
+Run benchmark. By default, the result will be printed to stdout. You can use `--noprint-result` to
+suppress this output.
+
+Sample command:
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--conf /absolute_path/to/conf_{stageId}_{partitionId}_{vId}.ini \
+--plan /absolute_path/to/plan_{stageId}_{partitionId}_{vId}.json \
+--split /absolut_path/to/split_{stageId}_{partitionId}_{vId}_{splitIdx}.json,/absolut_path/to/split_{stageId}_{partitionId}_{vId}_{splitIdx}.json \
+--threads 1 --noprint-result
+
+# If the stage requires data files, use --data-file to specify the absolute path.
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--conf /absolute_path/to/conf_{stageId}_{partitionId}_{vId}.ini \
+--plan /absolute_path/to/plan_{stageId}_{partitionId}_{vId}.json \
+--split /absolut_path/to/split_{stageId}_{partitionId}_{vId}_{splitIdx}.json,/absolut_path/to/split_{stageId}_{partitionId}_{vId}_{splitIdx}.json \
+--data /absolut_path/to/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet,/absolut_path/to/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet \
+--threads 1 --noprint-result
+```
+
+If the simulated stage is a middle stage, which means pure shuffle stage, you will get 3 types of
+dumped file:
+
+- Configuration file: INI formatted, file name `conf_{stageId}_{partitionId}_{vId}.ini`. Contains the
+ configurations to init Velox backend and runtime session.
+- Plan file: JSON formatted, file name `plan_{stageId}_{partitionId}_{vId}.json`. Contains the substrait
+ plan to the stage.
+- Data file: Parquet formatted, file name `data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet`.
+ There can be more than one input data file in a middle stage task. The input data files of a
+ middle stage will be loaded as iterators to serve as the inputs for the pipeline:
+
+```
+"localFiles": {
+ "items": [
+ {
+ "uriFile": "iterator:0"
+ }
+ ]
+}
+```
+
+Sample command:
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--conf /absolute_path/to/conf_{stageId}_{partitionId}_{vId}.ini \
+--plan /absolute_path/to/plan_{stageId}_{partitionId}_{vId}.json \
+--data /absolut_path/to/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet,/absolut_path/to/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet \
+--threads 1 --noprint-result
+```
+
+For some complex queries, stageId may cannot represent the Substrait plan input, please get the
+taskId from spark UI, and get your target parquet from saveDir.
+
+In this example, only one partition input with stage id 3 partition id 2, partitionId is 36, iterator length is 2.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/plan_3_36_0.json \
+--data /absolute_path/to/data_3_36_0_0.parquet,/absolute_path/to/data_3_36_0_1.parquet \
+--threads 1 --noprint-result
+```
+
+## Save output to parquet for analysis
+
+You can save the output to a parquet file via `--save-output <output>`
+
+Note: 1. This option cannot be used together with `--with-shuffle`. 2. This option cannot be used
+for write tasks. Please refer to section [Simulate write tasks](#simulate-write-tasks) for more
+details.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/plan_{stageId}_{partitionId}_{vId}.json \
+--data /absolute_path/to/data_{stageId}_{partitionId}_{vId}_{iteratorIdx}.parquet
+--threads 1 --noprint-result --save-output /absolute_path/to/result.parquet
+```
+
+## Add shuffle write process
+
+You can add the shuffle write process at the end of the pipeline via `--with-shuffle`
+
+Note: 1. This option cannot be used together with `--save-output`. 2. This option cannot be used
+for write tasks. Please refer to section [Simulate write tasks](#simulate-write-tasks) for more
+details.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/plan_{stageId}_{partitionId}_{vId}.json \
+--split /absolute_path/to/split_{stageId}_{partitionId}_{vId}_{splitIdx}.json \
+--threads 1 --noprint-result --with-shuffle
+```
+
+Developers can leverage the `--with-shuffle` option to benchmark the shuffle-write process by creating
+a simple pipeline of `table scan + shuffle write` in Gluten. This can be done by dumping the micro benchmark
+inputs from a first stage. The steps are demonstrated as below:
+
+1. Start spark-shell or pyspark
+
+We need to set `spark.gluten.sql.benchmark_task.stageId` and `spark.gluten.saveDir` to dump the inputs.
+Normally, the stage id should be greater than 0. You can run the command in step 2 in advance to get the
+right stage id in your case. We shall set `spark.default.parallelism` to 1 and `spark.sql.files.maxPartitionBytes`
+large enough to make sure there will be only 1 task in the first stage.
+
+```
+# Start pyspark
+./bin/pyspark --master local[*] \
+--conf spark.gluten.sql.benchmark_task.stageId=1 \
+--conf spark.gluten.saveDir=/path/to/saveDir \
+--conf spark.default.parallelism=1 \
+--conf spark.sql.files.maxPartitionBytes=10g
+... # omit other spark & gluten config
+```
+
+2. Run the table-scan command to dump the plan for the first stage
+
+If simulating single or round-robin partitioning, the first stage can only have the table scan operator.
+
+```
+>>> spark.read.format("parquet").load("file:///example.parquet").show()
+```
+
+If simulating hash partitioning, there will be a projection for generating the hash partitioning key.
+Therefore we need to explicitly run the `repartition` to generate the `scan + project` pipeline for the first stage.
+Note that using different number of shuffle partitions here doesn't change the generated pipeline.
+
+```
+>>> spark.read.format("parquet").load("file:///example.parquet").repartition(10, "key1", "key2").show()
+```
+
+Simuating range partitioning is not supported.
+
+3. Run the micro benchmark with dumped inputs
+
+General configurations for shuffle write:
+
+- `--with-shuffle`: Add shuffle write process at the end of the pipeline
+- `--shuffle-writer`: Specify shuffle writer type. Valid options are sort and hash. Default is hash.
+- `--partitioning`: Specify partitioning type. Valid options are rr, hash and single. Defualt is rr.
+ The partitioning type should match the command in step 2.
+- `--shuffle-partitions`: Specify number of shuffle partitions.
+- `--compression`: By default, the compression codec for shuffle outputs is lz4. You can switch to other compression codecs
+ or use hardware accelerators Valid options are: lz4, zstd, qat-gzip, qat-zstd and iaa-gzip. The compression levels are fixed (use default compression level 1).
+
+ Note using QAT or IAA codec requires Gluten cpp is built with these features.
+ Please check the corresponding section in [Velox document](../get-started/Velox.md) first for how to
+ setup, build and enable these features in Gluten. For QAT support, please
+ check [Intel® QuickAssist Technology (QAT) support](../get-started/Velox.md#intel-quickassist-technology-qat-support).
+ For IAA support, please
+ check [Intel® In-memory Analytics Accelerator (IAA/IAX) support](../get-started/Velox.md#intel-in-memory-analytics-accelerator-iaaiax-support)
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /path/to/saveDir/plan_{stageId}_{partitionId}_{vId}.json \
+--conf /path/to/saveDir/conf_{stageId}_{partitionId}_{vId}.ini \
+--split /path/to/saveDir/split_{stageId}_{partitionId}_{vId}_{splitIdx}.json \
+--with-shuffle \
+--shuffle-writer sort \
+--partitioning hash \
+--threads 1
+```
+
+### Run shuffle write/read task only
+
+Developers can only run shuffle write task via specifying `--run-shuffle` and `--data` options.
+The parquet format input will be read from arrow-parquet reader and sent to shuffle writer.
+The `--run-shuffle` option is similar to the `--with-shuffle` option, but it doesn't require the plan and split files.
+The round-robin partitioner is used by default. Besides, random partitioning can be used for testing purpose.
+By specifying option `--partitioning random`, the partitioner will generate a random partition id for each row.
+To evaluate the shuffle reader performance, developers can set `--run-shuffle-read` option to add read process after the write task finishes.
+
+The below command will run shuffle write/read in single thread, using sort shuffle writer with 40000 partitions and random partition id.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--run-shuffle \
+--run-shuffle-read \
+--data /path/to/input_for_shuffle_write.parquet
+--shuffle-writer sort \
+--partitioning random \
+--shuffle-partitions 40000 \
+--threads 1
+```
+
+The output should be like:
+
+```
+-------------------------------------------------------------------------------------------------------------------------
+Benchmark Time CPU Iterations UserCounters...
+-------------------------------------------------------------------------------------------------------------------------
+ShuffleWriteRead/iterations:1/process_time/real_time/threads:1 121637629714 ns 121309450910 ns 1 elapsed_time=121.638G read_input_time=25.2637G shuffle_compress_time=10.8311G shuffle_decompress_time=4.04055G shuffle_deserialize_time=7.24289G shuffle_spill_time=0 shuffle_split_time=69.9098G shuffle_write_time=2.03274G
+```
+
+## Enable debug mode
+
+`spark.gluten.sql.debug`(debug mode) is set to false by default thereby the google glog levels are limited to only print `WARNING` or higher severity logs.
+Unless `spark.gluten.sql.debug` is set in the INI file via `--conf`, the logging behavior is same as debug mode off.
+Developers can use `--debug-mode` command line flag to turn on debug mode when needed, and set verbosity/severity level via command line flags `--v` and `--minloglevel`. Note that constructing and deconstructing log strings can be very time-consuming, which may cause benchmark times to be inaccurate.
+
+
+## Enable HDFS support
+
+After enabling the dynamic loading of libhdfs.so at runtime to support HDFS, if you run the benchmark with an HDFS file, you need to set the classpath for Hadoop. You can do this by running
+```
+export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`
+```
+Otherwise, the HDFS connection will fail. If you have replaced ${HADOOP_HOME}/lib/native/libhdfs.so with libhdfs3.so, there is no need to set the `CLASSPATH`.
+
+## Simulate write tasks
+
+The last operator for a write task is a file write operator, and the output from Velox pipeline only
+contains several columns of statistics data. Therefore, specifying
+options `--with-shuffle` and `--save-output` does not take effect. You can specify the output path
+for the writer via `--write-path` option. Default is /tmp.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/plan.json \
+--split /absolute_path/to/split.json \
+--write-path /absolute_path/<dir>
+```
+
+## Simulate task spilling
+
+You can simulate task spilling by specify a memory hard limit from `--memory_limit`. By default, spilled files are written to the `/tmp` directory.
+To simulate real Gluten workloads, which utilize multiple spill directories, set the environment variable GLUTEN_SPARK_LOCAL_DIRS to a comma-separated string.
+Please check [Simulate Gluten workload with multiple processes and threads](#Simulate-Gluten-workload-with-multiple-processes-and-threads) for more details.
+
+## Simulate Gluten workload with multiple processes and threads
+
+You can use below command to launch several processes and threads to simulate parallel execution on
+Spark. Each thread in the same process will be pinned to the core number starting from `--cpu`.
+
+Suppose running on a bare-metal machine with 48C, 2-socket, HT-on, launching below command will
+utilize all vcores.
+
+```shell
+processes=24 # Same value of spark.executor.instances
+threads=8 # Same value of spark.executor.cores
+
+for ((i=0; i<${processes}; i++)); do
+ ./generic_benchmark --plan /path/to/plan.json --split /path/to/split.json --noprint-result --threads $threads --cpu $((i*threads)) &
+done
+```
+
+To include the shuffle write process or trigger spilling via `--memory-limit`,
+you can specify multiple directories by setting the `GLUTEN_SPARK_LOCAL_DIRS` environment variable
+to a comma-separated string. This will distribute the I/O load across multiple disks, similar to how it works for Gluten workloads.
+Temporary subdirectories will be created under each specified directory at runtime and will be automatically deleted if the process completes normally.
+
+```shell
+mkdir -p {/data1,/data2,/data3}/tmp # Make sure each directory has been already created.
+export GLUTEN_SPARK_LOCAL_DIRS=/data1/tmp,/data2/tmp,/data3/tmp
+
+processes=24 # Same value of spark.executor.instances
+threads=8 # Same value of spark.executor.cores
+
+for ((i=0; i<${processes}; i++)); do
+ ./generic_benchmark --plan /path/to/plan.json --split /path/to/split.json --noprint-result --with-shuffle --threads $threads --cpu $((i*threads)) &
+done
+```
+
+### Run Examples
+
+We also provide some example inputs in [cpp/velox/benchmarks/data](../../cpp/velox/benchmarks/data).
+E.g. Files under [generic_q5](../../cpp/velox/benchmarks/data/generic_q5) simulates a first-stage in
+TPCH Q5, which has a heavy table scan. You can follow below steps to run this example.
+
+1.
+
+Open [generic_q5/q5_first_stage_0_split.json](../../cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json)
+with file editor. Search for `"uriFile": "LINEITEM"` and replace `LINEITEM` with the URI to one
+partition file in lineitem. In the next line, replace the number in `"length": "..."` with the
+actual file length. Suppose you are using the provided small TPCH table
+in [cpp/velox/benchmarks/data/tpch_sf10m](../../cpp/velox/benchmarks/data/tpch_sf10m), the replaced
+JSON should be like:
+
+```
+{
+ "items": [
+ {
+ "uriFile": "file:///path/to/gluten/cpp/velox/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet",
+ "length": "1863237",
+ "parquet": {}
+ }
+ ]
+}
+```
+
+2. Launch multiple processes and multiple threads. Set `GLUTEN_SPARK_LOCAL_DIRS` and add
+ `--with-shuffle` to the command.
+
+```
+mkdir -p {/data1,/data2,/data3}/tmp # Make sure each directory has been already created.
+export GLUTEN_SPARK_LOCAL_DIRS=/data1/tmp,/data2/tmp,/data3/tmp
+
+processes=24 # Same value of spark.executor.instances
+threads=8 # Same value of spark.executor.cores
+
+for ((i=0; i<${processes}; i++)); do
+ ./generic_benchmark --plan /path/to/gluten/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0.json --split /path/to/gluten/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json --noprint-result --with-shuffle --threads $threads --cpu $((i*threads)) &
+done >stdout.log 2>stderr.log
+```
+
+You can find the "elapsed_time" and other metrics in stdout.log. In below output, the "elapsed_time"
+is ~10.75s. If you run TPCH Q5 with Gluten on Spark, a single task in the same Spark stage should
+take about the same time.
+
+```
+------------------------------------------------------------------------------------------------------------------
+Benchmark Time CPU Iterations UserCounters...
+------------------------------------------------------------------------------------------------------------------
+SkipInput/iterations:1/process_time/real_time/threads:8 1317255379 ns 10061941861 ns 8 collect_batch_time=0 elapsed_time=10.7563G shuffle_compress_time=4.19964G shuffle_spill_time=0 shuffle_split_time=0 shuffle_write_time=1.91651G
+```
+
+
diff --git a/_docs/v1.5.0/developers/NewToGluten.md b/_docs/v1.5.0/developers/NewToGluten.md
new file mode 100644
index 0000000..46218d6
--- /dev/null
+++ b/_docs/v1.5.0/developers/NewToGluten.md
@@ -0,0 +1,390 @@
+---
+layout: page
+title: NewToGluten
+nav_order: 4
+parent: Developers
+grand_parent: v1.5.0
+---
+# Guide for New Developers
+
+## Environment
+
+Gluten supports Ubuntu 20.04/22.04, CentOS 7/8, and MacOS.
+
+### JDK
+
+Gluten supports JDK 8 for Spark 3.2, 3.3, 3.4, and 3.5. For Spark 3.3 and later versions, Gluten
+also supports JDK 11 and 17.
+
+Note: Starting with Spark 4.0, the minimum required JDK version is 17.
+We recommend using a higher JDK version now to ease migration when deploying Gluten for Spark 4.0
+in the future. In addition, we may upgrade Arrow from 15.0.0 to a newer release, which will require
+JDK 11 as the minimum version.
+
+By default, Gluten compiles packages using JDK 8. Enable maven profile by `-Pjava-17` or `-Pjava-11` to use the corresponding JDK version, and ensure that the JDK version is available in your environment.
+
+If JDK 11 or a higher version is used, Spark and Arrow require setting the java args `-Dio.netty.tryReflectionSetAccessible=true`, see [SPARK-29924](https://issues.apache.org/jira/browse/SPARK-29924) and [ARROW-6206](https://issues.apache.org/jira/browse/ARROW-6206).
+
+Add the following configs in `spark-defaults.conf`:
+
+```
+spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
+spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
+```
+
+### Maven
+
+Gluten requires Maven 3.6.3 or above.
+
+### GCC
+
+Gluten requires GCC 11 or above.
+
+## Development
+
+To debug Java/Scala code, follow the steps in [build-gluten-with-velox-backend](../get-started/Velox.md#build-gluten-with-velox-backend).
+
+To debug C++ code, compile the backend code and gluten C++ code in debug mode.
+
+```bash
+## compile Velox backend with benchmark and tests to debug
+gluten_home/dev/builddeps-veloxbe.sh --build_tests=ON --build_benchmarks=ON --build_type=Debug
+```
+
+Note: To debug the tests under `<gluten_home>/gluten-ut/`, you must compile java code with `-Pspark-ut`.
+
+### Java/scala code development
+
+#### Linux IntelliJ local debug
+
+Install the Linux IntelliJ version, and debug code locally.
+
+- Ask your linux maintainer to install the desktop, and then restart the server.
+- If you use Moba-XTerm to connect, you don't need to install x11 server. If you are using another tool, such as putty, follow this guide:
+ [X11 Forwarding: Setup Instructions for Linux and Mac](https://www.businessnewsdaily.com/11035-how-to-use-x11-forwarding.html)
+
+- Download [IntelliJ Linux community version](https://www.jetbrains.com/idea/download/?fromIDE=#section=linux) to Linux server.
+- Start Idea using the following command:
+
+ `bash <idea_dir>/idea.sh`
+
+#### Set up Gluten project
+
+- Make sure you have compiled Gluten.
+- Load the Gluten by **File**->**Open**, select **<gluten_home/pom.xml>**.
+- Activate your profiles such as `<backends-velox>`, then **Reload Maven Project** to activate all the needed modules.
+- Create breakpoints and debug as you wish. You can use `CTRL+N` to locate a test class to start your test.
+
+#### Java/Scala code style
+
+IntelliJ supports importing settings for Java/Scala code style. You can import [intellij-codestyle.xml](../../dev/intellij-codestyle.xml) to your IDE.
+See [IntelliJ guide](https://www.jetbrains.com/help/idea/configuring-code-style.html#import-code-style).
+
+To format Java/Scala code using the [Spotless](https://github.com/diffplug/spotless) plugin, run the following command:
+
+```
+./dev/format-scala-code.sh
+```
+
+### C++ code development
+
+This guide is for remote debugging by connecting to the remote Linux server using `SSH`.
+
+Download and install [Visual Studio Code](https://code.visualstudio.com/Download).
+
+Key components found on the left side bar are:
+- Explorer (Project structure)
+- Search
+- Run and Debug
+- Extensions (Install the C/C++ Extension Pack, Remote Development, and GitLens. C++ Test Mate is also suggested.)
+- Remote Explorer (To connect to the linux server using ssh, click **+**, then enter `ssh USERNAME@REMOTE_SERVER_IP_ADDRESS`)
+- Manage (Settings)
+
+Input your password in the above pop-up window. It will take a few minutes to install the Linux VSCode server in the folder `~/.vscode-server` on the remote machine.
+
+If the download fails, delete this folder and try again.
+
+Note: If VSCode is upgraded, you must download the linux server again. We recommend switching the update mode to `off`. Search `update` in **Manage**->**Settings** to turn off update mode.
+
+#### Set up project
+
+- Select **File**->**Open Folder**, then select the Gluten folder.
+- After the project loads, you will be prompted to **Select CMakeLists.txt**. Select the
+ `${workspaceFolder}/cpp/CMakeLists.txt` file.
+- Next, you will be prompted to **Select a Kit** for the Gluten project. Select **GCC 11** or above.
+
+#### Settings
+
+VSCode supports two ways to configure user settings.
+
+- **Manage**->**Command Palette** (Open `settings.json`, search by `Preferences: Open Settings (JSON)`)
+- **Manage**->**Settings** (Common setting)
+
+#### Build using VSCode
+
+VSCode will try to compile using debug mode in `<gluten_home>/build`. You must compile Velox debug mode before
+compiling Gluten.
+
+Note: If you have previously compiled Velox in release mode, use the command below to compile in debug mode.
+
+```bash
+cd incubator-gluten/ep/build-velox/build/velox_ep
+
+# Build the Velox debug version in <velox_home>/_build/debug
+make debug EXTRA_CMAKE_FLAGS="-DVELOX_ENABLE_PARQUET=ON -DENABLE_HDFS=ON -DVELOX_BUILD_TESTING=OFF -DVELOX_ENABLE_DUCKDB=ON -DVELOX_BUILD_TEST_UTILS=ON"
+```
+
+Then Gluten will link the Velox debug library.
+
+Click **build** in the bottom bar to enable IntelliSense features like search and navigation.
+
+#### Debug setting
+
+The default compile command does not enable tests and benchmarks, so the corresponding executable files are not generated.
+To enable the test and benchmark args, create or edit the `<gluten_home>/.vscode/settings.json` to add the
+configurations below:
+
+```json
+{
+ "cmake.configureArgs": [
+ "-DBUILD_BENCHMARKS=ON",
+ "-DBUILD_TESTS=ON"
+ ],
+ "C_Cpp.default.configurationProvider": "ms-vscode.cmake-tools"
+}
+```
+
+After compiling with these updated configs, you should have executable files, such as
+`<gluten_home>/cpp/build/velox/tests/velox_shuffle_writer_test`.
+
+Open the **Run and Debug** panel (Ctrl-Shift-D) and then click the link to create a `launch.json` file. If prompted,
+select a debugger like **C++ (GDB/LLDB)**. The `launch.json` will be created under `<gluten_home>/.vscode/`.
+
+Note: Change `name`, `program`, `args` for your environment.
+
+Click the **Add Configuration** button in `launch.json`, and select gdb **launch** to start a program for debugging or
+**attach** to attach a running program for debugging.
+
+Then you can create breakpoints and debug using **Run and Debug** in Visual Studio Code.
+
+#### Debug Velox code
+
+For some Velox tests such as `ParquetReaderTest`, tests need to read the parquet file in `<velox_home>/velox/dwio/parquet/tests/examples`.
+Select `ParquetReaderTest.cpp` in the IDE window, then click **Start Debugging**, otherwise `No such file or directory` exception will be raised.
+
+#### Clang format
+
+Gluten uses clang-format 15 to format source files.
+
+```bash
+apt-get install clang-format-15
+```
+
+Set config in `settings.json`
+
+```json
+"clang-format.executable": "clang-format-15",
+"editor.formatOnSave": true,
+```
+
+If multiple clang-format versions are installed, `formatOnSave` may not take effect. To specify the default formatter,
+search for `default formatter` in **Settings**, then select **Clang-Format**.
+
+If `formatOnSave` still has no effect, select a single file and use `SHIFT+ALT+F` to format it manually.
+
+#### CMake format
+
+To format cmake files like `CMakeLists.txt` and `*.cmake`, install `cmake-format`.
+```
+pip3 install --user cmake-format
+```
+Here is an example of how to format a file using the command line:
+```
+cmake-format --first-comment-is-literal True --in-place cpp/velox/CMakeLists.txt
+```
+
+After the above installation, you can optionally do some configuration in Visual Studio Code to easily format cmake files.
+1. Install `cmake-format` extension in Visual Studio Code.
+2. Configure the extension. To do this, open the settings (**File** -> **Preferences** -> **Settings**), search for `cmake-format`,
+ and configure the following settings as shown:
+ * Set **Args**: `--first-comment-is-literal=True`.
+ * Set **Exe Path** to the path of the `cmake-format` command. If you installed `cmake-format` in a standard
+ location, you might not need to change this setting.
+3. Format your CMake files by right-clicking in a file and selecting `Format Document`.
+
+#### Add unit tests
+
+1. For Native Code Modifications: If you have modified native code, use gtest to test the native code.
+ A secondary option is to add Gluten UT to ensure coverage.
+
+2. For Gluten-Related Code Modifications: If you have modified code related to Gluten, it is preferable to add scalatest rather than JUnit.
+ Additionally, the test classes should be placed in the org.apache.gluten package.
+
+3. For Spark-Related Code Modifications: If you have modified code related to Spark, it is preferable to add scalatest rather than JUnit.
+ Additionally, the test classes should be placed in the org.apache.spark package.
+
+4. Placement of Non-Native Code UTs: Ensure that unit tests for non-native code are placed within org.apache.gluten and org.apache.spark packages.
+ This is important because the CI system runs unit tests from these two paths in parallel. Placing tests in other paths might cause your tests to be ignored.
+
+#### View Surefire reports of Scala unit tests in GHA
+
+Surefire reports are invaluable tools in the ecosystem of Java-based applications that utilize the Maven build automation tool.
+These reports are generated by the Maven Surefire Plugin during the testing phase of your build process.
+They compile results from unit tests, providing detailed insights into which tests passed or failed, what errors were encountered, and other essential metrics.
+
+Surefire reports play a crucial role in the development and maintenance of high-quality software.
+In GitHub Actions, we expose Surefire test reports so developers can review error messages and stack traces from failing unit tests.
+
+To check Surefire reports:
+
+1. Click the **Checks** Tab in PR.
+2. Find **Report test results** in **Dev PR**.
+3. There, you can check the results with summary and annotations.
+
+
+
+## Debug C++ Code with Core Dump
+
+```bash
+mkdir -p /mnt/DP_disk1/core
+sysctl -w kernel.core_pattern=/mnt/DP_disk1/core/core-%e-%p-%t
+cat /proc/sys/kernel/core_pattern
+# set the core file to unlimited size
+echo "ulimit -c unlimited" >> ~/.bashrc
+# then you will get the core file at `/mnt/DP_disk1/core` when the program crashes
+# gdb -c corefile
+# gdb <gluten_home>/cpp/build/releases/libgluten.so 'core-Executor task l-2000883-1671542526'
+```
+
+`core-Executor task l-2000883-1671542526` is the generated core file name.
+
+```bash
+(gdb) bt
+(gdb) f7
+(gdb) set print pretty on
+(gdb) p *this
+```
+
+- Get the backtrace
+- Switch to 7th stack
+- Print the variable in a more readable way
+- Print the variable fields
+
+Sometimes you only get the C++ exception message. If that happens, you can generate a core dump file by running the following code:
+```cpp
+char* p = nullptr;
+*p = 'a';
+```
+or by the following commands:
+- `gcore <pid>`
+- `kill -s SIGSEGV <pid>`
+
+## Debug C++ with GDB
+
+You can use GDB to debug tests, benchmarks, and JNI calls.
+Place the following code to your debug path.
+
+```cpp
+pid_t pid = getpid();
+printf("----------------------------------pid: %lun", pid);
+sleep(10);
+```
+
+You can also get the pid by java command or grep java program when executing unit test.
+
+```bash
+jps
+1375551 ScalaTestRunner
+ps ux | grep TestOperator
+```
+
+Execute GDB command to debug:
+```bash
+gdb attach <pid>
+```
+
+```bash
+gdb attach 1375551
+wait to attach....
+(gdb) b <velox_home>/velox/substrait/SubstraitToVeloxPlan.cpp:577
+(gdb) c
+```
+
+## Debug Memory Leaks
+
+### Arrow memory allocator leak
+
+If you receive an error message like the following:
+
+```bash
+4/04/18 08:15:38 WARN ArrowBufferAllocators$ArrowBufferAllocatorManager: Detected leaked Arrow allocator [Default], size: 191, process accumulated leaked size: 191...
+24/04/18 08:15:38 WARN ArrowBufferAllocators$ArrowBufferAllocatorManager: Leaked allocator stack Allocator(ROOT) 0/191/319/9223372036854775807 (res/actual/peak/limit)
+```
+You can open the Arrow allocator debug config by adding the VP option `-Darrow.memory.debug.allocator=true`. That gives you more details, like the following example:
+
+```bash
+child allocators: 0
+ ledgers: 7
+ ledger[10] allocator: ROOT), isOwning: , size: , references: 1, life: 10483701311283711..0, allocatorManager: [, life: ] holds 1 buffers.
+ ArrowBuf[11], address:140100698555856, capacity:128
+ event log for: ArrowBuf[11]
+ 10483701311362601 create()
+ at org.apache.arrow.memory.util.HistoricalLog$Event.<init>(HistoricalLog.java:175)
+ at org.apache.arrow.memory.util.HistoricalLog.recordEvent(HistoricalLog.java:83)
+ at org.apache.arrow.memory.ArrowBuf.<init>(ArrowBuf.java:97)
+ at org.apache.arrow.memory.BufferLedger.newArrowBuf(BufferLedger.java:271)
+ at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:340)
+ at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:316)
+ at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
+ at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:280)
+ at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)
+ at org.apache.arrow.c.ArrowArray.allocateNew(ArrowArray.java:116)
+ at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:61)
+ at org.apache.arrow.c.Data.importIntoVector(Data.java:289)
+ at org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:332)
+ at org.apache.arrow.dataset.jni.NativeScanner$NativeReader.loadNextBatch(NativeScanner.java:151)
+ at org.apache.gluten.datasource.ArrowFileFormat$$anon$1.hasNext(ArrowFileFormat.scala:99)
+ at org.apache.gluten.utils.IteratorCompleter.hasNext(Iterators.scala:69)
+ at org.apache.spark.memory.SparkMemoryUtil$UnsafeItr.hasNext(SparkMemoryUtil.scala:246)
+```
+
+### CPP code memory leak
+
+Sometimes you cannot get the coredump symbols when debugging a memory leak. You can write a GoogleTest to use valgrind for detection.
+
+```bash
+apt install valgrind
+valgrind --leak-check=yes ./exec_backend_test
+```
+
+## Run TPC-H and TPC-DS
+
+We supply `<gluten_home>/tools/gluten-it` to execute these queries.
+See [velox_backend_x86.yml](https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_backend_x86.yml).
+
+## Enable Gluten for Spark
+
+To enable Gluten Velox backend for Spark, run the following command:
+```
+spark-shell --name run_gluten \
+ --master yarn --deploy-mode client \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=20g \
+ --jars https://dlcdn.apache.org/incubator/gluten/1.4.0-incubating/apache-gluten-1.4.0-incubating-bin-spark35.tar.gz \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
+```
+
+## Gluten Plan Validation and Updates
+
+`VeloxTPCHSuite` can verify the executed Gluten plans for the TPC-H benchmark to avoid unintentional changes.
+This verification is based on comparisons with the golden files that record the expected Gluten plans.
+
+The following failure may occur in GitHub CI or local tests:
+```log
+- TPC-H q5 *** FAILED ***
+ Mismatch for query 5
+ Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
+ Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)
+```
+To update the golden files, find the actual Gluten plans in GitHub CI Artifacts or the local `/tmp/` directory, then update the corresponding golden files in the `tpch-approved-plan/` directory.
diff --git a/_docs/v1.5.0/developers/PartialProject.md b/_docs/v1.5.0/developers/PartialProject.md
new file mode 100644
index 0000000..c715ea1
--- /dev/null
+++ b/_docs/v1.5.0/developers/PartialProject.md
@@ -0,0 +1,30 @@
+---
+layout: page
+title: PartialProject
+nav_order: 5
+parent: Developers
+grand_parent: v1.5.0
+---
+# Partial Projection Support
+
+In Gluten, there is still a gap in supporting all Spark expressions natively (e.g., some JSON functions or Java UDFs). In this case, Gluten will choose the JVM code path to run the expressions, which can introduce performance regressions.
+
+Partial projections, which allow Gluten to minimal data copy between JVM and C++, were added to avoid these performance regressions.
+
+
+## Detailed Implementations
+
+### Adding Partial Projection for UDF
+
+For example, with the expression `hash(udf(col0)), col1, col2, col3, col4`, partial projection allows us to convert only `col0` to row or column to Arrow as input, and convert `udf(col0)` as an alias `partialProject1_`. Then, ProjectExecTransformer will handle `hash(partialProject1_), col1, col2, col3, col4, partialProject1_`. This feature saves the cost of converting the columnar format to row format and vice-versa.
+
+
+## Adding Partial Projection for Unsupported Expressions
+
+The partial projection feature can also benefit from expressions that are not natively supported. For example, `substr(from_json(col_a))`. Since `from_json` is not fully supported, Gluten may use the JVM code path. Instead of projecting the whole expression, partial projection will attempt to project `from_json()` and perform a native projection of `substr()`.
+
+In the case of blacklisted expressions defined in `spark.gluten.expression.blacklist`, this feature is also beneficial.
+
+## Limitations
+
+This feature is in the preliminary stages of development and will be improved in future updates.
\ No newline at end of file
diff --git a/_docs/v1.5.0/developers/ProfileMemoryOfGlutenWithVelox.md b/_docs/v1.5.0/developers/ProfileMemoryOfGlutenWithVelox.md
new file mode 100644
index 0000000..ff60242
--- /dev/null
+++ b/_docs/v1.5.0/developers/ProfileMemoryOfGlutenWithVelox.md
@@ -0,0 +1,159 @@
+---
+layout: page
+title: ProfileMemoryOfGlutenWithVelox
+nav_order: 6
+parent: Developers
+grand_parent: v1.5.0
+---
+Gluten offloads most of Spark SQL execution to native engine. We can use [gperftools](https://github.com/gperftools/gperftools) or [jemalloc](https://github.com/jemalloc/jemalloc)
+to analyze the offheap memory and cpu profile.
+
+# Profile with gperftools
+
+`gperftools` is a collection of a high-performance multi-threaded
+malloc() implementation, plus some pretty nifty performance analysis
+tools, see more: https://github.com/gperftools/gperftools/wiki.
+
+## Build and install gperftools
+
+Download `gperftools` from https://github.com/gperftools/gperftools/releases, build and install.
+
+```bash
+wget https://github.com/gperftools/gperftools/releases/download/gperftools<version>/gperftools-<version>.tar.gz
+tar xzvf gperftools-<version>.tar.gz
+cd gperftools-<version>
+./configure
+make && make install
+```
+
+Then we can find the tcmalloc libraries in `$GPERFTOOLS_HOME/.lib`.
+
+## Run Gluten with gperftools
+
+Configure `--files` or `spark.files` for Spark.
+
+```
+--files /path/to/gperftools/libtcmalloc_and_profiler.so
+or
+spark.files /path/to/gperftools/libtcmalloc_and_profiler.so
+```
+
+Use `LD_PRELOAD` to preload tcmalloc library, and enable heap profile with `HEAPPROFILE` or cpu profile with `CPUPROFILE`.
+
+Example of enabling heap profile in spark executor:
+
+```
+spark.executorEnv.LD_PRELOAD ./libtcmalloc_and_profiler.so
+
+# Specifies dump profile path. ${CONTAINER_ID} is only used to distinguish the result files when running on yarn.
+spark.executorEnv.HEAPPROFILE /tmp/gluten_heap_perf_${CONTAINER_ID}
+```
+
+Finally, profiling files prefixed with `/tmp/gluten_heap_perf_${CONTAINER_ID}` will be generated for each spark executor.
+
+## Analyze profiling output
+
+Prepare the required native libraries. Assume static build is used for Gluten, there is no other shared dependency libs.
+
+```bash
+jar xf gluten-velox-bundle-spark3.5_2.12-centos_7_x86_64-1.2.0.jar relative/path/to/libvelox.so ralative/path/to/libgluten.so
+mv libvelox.so libgluten.so /path/to/gluten_lib_prefix
+```
+
+Generate a GIF of the analysis result:
+
+```bash
+# `/usr/bin/java` indicates the program used by running spark executor
+pprof --show_bytes --gif --lib_prefix=/path/to/gluten_lib_prefix /usr/bin/java /path/to/gluten_heap_perf_XXX > result.gif
+```
+
+Result like:
+
+<img src="../image/velox_profile_memory_gif.gif" width="200" />
+
+Or display analysis result in TEXT:
+
+```bash
+pprof --text --lib_prefix=/path/to/gluten_lib_prefix /usr/bin/java /path/to/gluten_heap_perf_XXX
+```
+
+Result like:
+
+<img src="../image/velox_profile_memory_text.png" width="400" />
+
+**\*\*** Get more help from https://github.com/gperftools/gperftools/wiki#documentation.
+
+# Profile with jemalloc
+
+`jemalloc` is a general purpose malloc(3) implementation that emphasizes fragmentation
+avoidance and scalable concurrency support. We can also use it to analyze Gluten performance.
+Getting Started with `jemalloc`: https://github.com/jemalloc/jemalloc/wiki/Getting-Started.
+
+## Build and install jemalloc
+
+Download `jemalloc` from https://github.com/jemalloc/jemalloc/releases, build and install.
+
+```
+cd /path/to/jemalloc
+./autogen.sh --enable-prof
+make && make install
+```
+Then we can find the jemalloc library in `$JEMALLOC_HOME/.lib`.
+
+## Run Gluten with jemalloc
+
+Configure `--files` or `spark.files` for Spark.
+
+```
+--files /path/to/jemalloc/libjemalloc.so
+or
+spark.files /path/to/jemalloc/libjemalloc.so
+```
+
+Example of enabling heap profile in spark executor:
+
+```
+spark.executorEnv.LD_PRELOAD ./libjemalloc.so
+spark.executorEnv.MALLOC_CONF prof:true,lg_prof_interval:30,prof_prefix:/tmp/gluten_heap_perf
+```
+
+Finally, profiling files prefixed with `/tmp/gluten_heap_perf.${PID}` will be generated for each spark executor.
+
+## Memory dump on spark executor exit
+
+Sometimes, when native memory is not managed by gluten or there are some memory leaks that will cause spark executor to be killed due to memory limit,
+we only need to trigger a memory dump on executor exit.
+
+If we want to enable this feature we need to follow steps:
+
+1. Build gluten with `--enable_jemalloc_stats=ON` to enabled jemalloc stats.
+2. Enabled memory dump on exit, add spark executor environments to load jemalloc lib and make memory profiling active.
+ ```
+ spark.gluten.monitor.memoryDumpOnExit=true
+ spark.executorEnv.LD_PRELOAD=/path/to/libjemalloc.so
+ spark.executorEnv.MALLOC_CONF=prof:true,prof_prefix:/tmp/gluten_heap_perf
+ ```
+
+## Analyze profiling output
+
+Prepare the required native libraries. Assume static build is used for Gluten, so there is no other shared dependency libs.
+
+```bash
+jar xf gluten-velox-bundle-spark3.5_2.12-centos_7_x86_64-1.2.0.jar relative/path/to/libvelox.so relative/path/to/libgluten.so
+mv libvelox.so libgluten.so /path/to/gluten_lib_prefix
+```
+
+Generate a GIF of the analysis result:
+
+```bash
+# `/usr/bin/java` indicates the program used by running spark executor
+jeprof --show_bytes --gif --lib_prefix=/path/to/gluten_lib_prefix /usr/bin/java /path/to/gluten_heap_perf_XXX > result.gif
+```
+
+Or display analysis result in TEXT:
+
+```bash
+jeprof --text --lib_prefix=/path/to/gluten_lib_prefix /usr/bin/java /path/to/gluten_heap_perf_XXX
+```
+
+**\*\*** Get more help from https://jemalloc.net/jemalloc.3.html.
diff --git a/_docs/v1.5.0/developers/QueryTrace.md b/_docs/v1.5.0/developers/QueryTrace.md
new file mode 100644
index 0000000..2d50f0a
--- /dev/null
+++ b/_docs/v1.5.0/developers/QueryTrace.md
@@ -0,0 +1,173 @@
+---
+layout: page
+title: QueryTrace
+nav_order: 7
+parent: Developers
+grand_parent: v1.5.0
+---
+# Background
+Currently, we have [MicroBenchmarks](https://github.com/apache/incubator-gluten/blob/main/docs/developers/MicroBenchmarks.md) to profile the Velox plan execution in stage level, now we have query trace replayer to specify plan node id to profile in operator level.
+
+We cannot profile operator level directly because ValueStreamNode cannot serialize and deserialize, so we should generate benchmark first, and then enable query trace in benchmark which replaces the ValueStreamNode to ValuesNode.
+
+Relevant link:
+https://facebookincubator.github.io/velox/develop/debugging/tracing.html
+
+https://facebookincubator.github.io/velox/configs.html#tracing
+
+https://velox-lib.io/blog/velox-query-tracing
+
+# Usage
+Firstly, we should use MicroBenchmark to generate the input parquet data file and substrait json plan.
+
+```shell
+/tmp/saveDir
+├── conf_35_0.ini
+├── data_35_0_0.parquet
+├── plan_35_0.json
+└── split_35_0_0.json
+```
+Secondly, execute the benchmark to get the task id and query trace node id, for example, the task id is `Gluten_Stage_0_TID_0_VTID_0` and we will profile partial aggregation of node id `7`.
+
+```shell
+W0113 15:30:18.903225 2524110 GenericBenchmark.cc:251] Setting CPU for thread 0 to 0
+I0113 15:30:19.093111 2524110 Task.cpp:2051] Terminating task Gluten_Stage_0_TID_0_VTID_0 with state Finished after running for 142ms
+W0113 15:30:19.093384 2524110 GenericBenchmark.cc:485] {Task Gluten_Stage_0_TID_0_VTID_0 (Gluten_Stage_0_TID_0_VTID_0) Finished
+Plan:
+-- Project[8][expressions: (n8_3:INTEGER, hash_with_seed(42,"n6_5","n6_6")), (n8_4:DOUBLE, "n6_5"), (n8_5:DOUBLE, "n6_6"), (n8_6:BIGINT, "n7_2")] -> n8_3:INTEGER, n8_4:DOUBLE, n8_5:DOUBLE, n8_6:BIGINT
+ -- Aggregation[7][PARTIAL [n6_5, n6_6] n7_2 := sum_partial("n6_7")] -> n6_5:DOUBLE, n6_6:DOUBLE, n7_2:BIGINT
+ -- Project[6][expressions: (n6_5:DOUBLE, "n5_7"), (n6_6:DOUBLE, "n5_8"), (n6_7:BIGINT, unscaled_value("n5_9"))] -> n6_5:DOUBLE, n6_6:DOUBLE, n6_7:BIGINT
+ -- Project[5][expressions: (n5_7:DOUBLE, "n1_4"), (n5_8:DOUBLE, "n1_5"), (n5_9:DECIMAL(7, 2), "n1_6"), (n5_10:DOUBLE, "n1_7"), (n5_11:DOUBLE, "n3_1")] -> n5_7:DOUBLE, n5_8:DOUBLE, n5_9:DECIMAL(7, 2), n5_10:DOUBLE, n5_11:DOUBLE
+ -- HashJoin[4][INNER n1_8=n3_2] -> n1_4:DOUBLE, n1_5:DOUBLE, n1_6:DECIMAL(7, 2), n1_7:DOUBLE, n1_8:DOUBLE, n3_1:DOUBLE, n3_2:DOUBLE
+ -- Project[1][expressions: (n1_4:DOUBLE, "n0_0"), (n1_5:DOUBLE, "n0_1"), (n1_6:DECIMAL(7, 2), "n0_2"), (n1_7:DOUBLE, "n0_3"), (n1_8:DOUBLE, "n0_3")] -> n1_4:DOUBLE, n1_5:DOUBLE, n1_6:DECIMAL(7, 2), n1_7:DOUBLE, n1_8:DOUBLE
+ -- TableScan[0][table: hive_table, remaining filter: (isnotnull("sr_store_sk"))] -> n0_0:DOUBLE, n0_1:DOUBLE, n0_2:DECIMAL(7, 2), n0_3:DOUBLE
+ -- Project[3][expressions: (n3_1:DOUBLE, "n2_0"), (n3_2:DOUBLE, "n2_0")] -> n3_1:DOUBLE, n3_2:DOUBLE
+ -- ValueStream[2][] -> n2_0:DOUBLE
+```
+
+Thirdly, execute the benchmark and enable the query trace.
+
+```shell
+ ./generic_benchmark --conf /tmp/saveDir/conf_35_0.ini --plan /tmp/saveDir/plan_35_0.json \ --split /tmp/saveDir/split_35_0_0.json --data /tmp/saveDir/data_35_0_0.parquet --query_trace_enabled=true --query_trace_dir=/tmp/query_trace --query_trace_node_ids=7 --query_trace_max_bytes=100000000000 --query_trace_task_reg_exp=Gluten_Stage_0_TID_0_VTID_0 --threads 1 --noprint-result
+```
+
+We can see the following message, which indicates the query trace runs successfully.
+```shell
+I0114 11:06:52.461263 2587302 Task.cpp:3072] Trace input for plan nodes 7 from task Gluten_Stage_0_TID_0_VTID_0
+I0114 11:06:52.462595 2587302 Operator.cpp:135] Trace input for operator type: PartialAggregation, operator id: 5, pipeline: 0, driver: 0, task: Gluten_Stage_0_TID_0_VTID_0
+```
+
+Now we can see the data in query trace directory `/tmp/query_trace`.
+
+```shell
+/tmp/query_trace/
+└── Gluten_Stage_0_TID_0_VTID_0
+ └── Gluten_Stage_0_TID_0_VTID_0
+ ├── 7
+ │ └── 0
+ │ └── 0
+ │ ├── op_input_trace.data
+ │ └── op_trace_summary.json
+ └── task_trace_meta.json
+```
+Fourthly, replay the query. Show the query trace summary by following command.
+
+```shell
+
+/mnt/DP_disk1/code/velox/build/velox/tool/trace# ./velox_query_replayer --root_dir /tmp/query_trace --task_id Gluten_Stage_0_TID_0_VTID_0 --query_id=Gluten_Stage_0_TID_0_VTID_0 --node_id=7 --summary
+WARNING: Logging before InitGoogleLogging() is written to STDERR
+I0115 20:27:25.821105 2684048 HiveConnector.cpp:56] Hive connector test-hive created with maximum of 20000 cached file handles.
+I0115 20:27:25.823112 2684048 TraceReplayRunner.cpp:223]
+++++++Query trace summary++++++
+Number of tasks: 1
+
+++++++Query configs++++++
+ max_spill_bytes: 107374182400
+ max_output_batch_rows: 4096
+ max_partial_aggregation_memory: 80530636
+ max_extended_partial_aggregation_memory: 120795955
+ max_spill_level: 4
+ spill_enabled: true
+ spark.bloom_filter.max_num_bits: 4194304
+ spillable_reservation_growth_pct: 25
+ query_trace_dir: /tmp/query_trace
+ spiller_num_partition_bits: 3
+ preferred_output_batch_rows: 4096
+ spill_write_buffer_size: 1048576
+ spiller_start_partition_bit: 29
+ spill_prefixsort_enabled: false
+ abandon_partial_aggregation_min_rows: 100000
+ adjust_timestamp_to_session_timezone: true
+ query_trace_max_bytes: 100000000000
+ query_trace_enabled: true
+ spark.legacy_date_formatter: false
+ join_spill_enabled: 1
+ query_trace_task_reg_exp: Gluten_Stage_0_TID_0_VTID_0
+ query_trace_node_ids: 7
+ order_by_spill_enabled: 1
+ aggregation_spill_enabled: 1
+ abandon_partial_aggregation_min_pct: 90
+ session_timezone: Etc/UTC
+ driver_cpu_time_slice_limit_ms: 0
+ max_spill_file_size: 1073741824
+ max_spill_run_rows: 3145728
+ spark.bloom_filter.expected_num_items: 1000000
+ spill_read_buffer_size: 1048576
+ spill_compression_codec: lz4
+ spark.partition_id: 0
+ max_split_preload_per_driver: 2
+ spark.bloom_filter.num_bits: 8388608
+
+++++++Connector configs++++++
+test-hive
+ file_column_names_read_as_lower_case: true
+ partition_path_as_lower_case: false
+ hive.reader.timestamp_unit: 6
+ hive.parquet.writer.timestamp_unit: 6
+ max_partitions_per_writers: 10000
+ ignore_missing_files: 0
+
+++++++Task query plan++++++
+-- Project[8][expressions: (n8_3:INTEGER, hash_with_seed(42,"n6_5","n6_6")), (n8_4:DOUBLE, "n6_5"), (n8_5:DOUBLE, "n6_6"), (n8_6:BIGINT, "n7_2")] -> n8_3:INTEGER, n8_4:DOUBLE, n8_5:DOUBLE, n8_6:BIGINT
+ -- Aggregation[7][PARTIAL [n6_5, n6_6] n7_2 := sum_partial("n6_7")] -> n6_5:DOUBLE, n6_6:DOUBLE, n7_2:BIGINT
+ -- Project[6][expressions: (n6_5:DOUBLE, "n5_7"), (n6_6:DOUBLE, "n5_8"), (n6_7:BIGINT, unscaled_value("n5_9"))] -> n6_5:DOUBLE, n6_6:DOUBLE, n6_7:BIGINT
+ -- Project[5][expressions: (n5_7:DOUBLE, "n1_4"), (n5_8:DOUBLE, "n1_5"), (n5_9:DECIMAL(7, 2), "n1_6"), (n5_10:DOUBLE, "n1_7"), (n5_11:DOUBLE, "n3_1")] -> n5_7:DOUBLE, n5_8:DOUBLE, n5_9:DECIMAL(7, 2), n5_10:DOUBLE, n5_11:DOUBLE
+ -- HashJoin[4][INNER n1_8=n3_2] -> n1_4:DOUBLE, n1_5:DOUBLE, n1_6:DECIMAL(7, 2), n1_7:DOUBLE, n1_8:DOUBLE, n3_1:DOUBLE, n3_2:DOUBLE
+ -- Project[1][expressions: (n1_4:DOUBLE, "n0_0"), (n1_5:DOUBLE, "n0_1"), (n1_6:DECIMAL(7, 2), "n0_2"), (n1_7:DOUBLE, "n0_3"), (n1_8:DOUBLE, "n0_3")] -> n1_4:DOUBLE, n1_5:DOUBLE, n1_6:DECIMAL(7, 2), n1_7:DOUBLE, n1_8:DOUBLE
+ -- TableScan[0][table: hive_table, remaining filter: (isnotnull("sr_store_sk"))] -> n0_0:DOUBLE, n0_1:DOUBLE, n0_2:DECIMAL(7, 2), n0_3:DOUBLE
+ -- Project[3][expressions: (n3_1:DOUBLE, "d_date_sk"), (n3_2:DOUBLE, "d_date_sk")] -> n3_1:DOUBLE, n3_2:DOUBLE
+ -- Values[2][366 rows in 1 vectors] -> d_date_sk:DOUBLE
+
+++++++Task Summaries++++++
+
+++++++Task Gluten_Stage_0_TID_0_VTID_0++++++
+
+++++++Pipeline 0++++++
+driver 0: opType PartialAggregation, inputRows 293762, inputBytes 5.69MB, rawInputRows 0, rawInputBytes 0B, peakMemory 5.39MB
+```
+Then you can use following command to re-execute the query plan.
+
+```shell
+/mnt/DP_disk1/code/velox/build/velox/tool/trace# ./velox_query_replayer --root_dir /tmp/query_trace --task_id Gluten_Stage_0_TID_0_VTID_0 --query_id=Gluten_Stage_0_TID_0_VTID_0 --node_id=7
+WARNING: Logging before InitGoogleLogging() is written to STDERR
+I0115 20:30:17.665169 2685397 HiveConnector.cpp:56] Hive connector test-hive created with maximum of 20000 cached file handles.
+I0115 20:30:17.676046 2685397 Cursor.cpp:192] Task spill directory[/tmp/velox_test_H163pi/test_cursor 1] created
+I0115 20:30:17.941792 2685398 Task.cpp:2051] Terminating task test_cursor 1 with state Finished after running for 265ms
+I0115 20:30:17.943285 2685397 OperatorReplayerBase.cpp:146] Stats of replaying operator PartialAggregation : Output: 292979 rows (9.27MB, 81 batches), Cpu time: 107.16ms, Wall time: 107.36ms, Blocked wall time: 0ns, Peak memory: 5.39MB, Memory allocations: 19, Threads: 1, CPU breakdown: B/I/O/F (252.36us/7.32ms/99.53ms/52.96us)
+I0115 20:30:17.943373 2685397 OperatorReplayerBase.cpp:149] Memory usage: Aggregation_replayer usage 0B reserved 0B peak 6.00MB
+ task.test_cursor 1 usage 0B reserved 0B peak 6.00MB
+ node.N/A usage 0B reserved 0B peak 0B
+ op.N/A.0.0.CallbackSink usage 0B reserved 0B peak 0B
+ node.1 usage 0B reserved 0B peak 6.00MB
+ op.1.0.0.PartialAggregation usage 0B reserved 0B peak 5.39MB
+ node.0 usage 0B reserved 0B peak 0B
+ op.0.0.0.OperatorTraceScan usage 0B reserved 0B peak 0B
+I0115 20:30:17.943809 2685397 TempDirectoryPath.cpp:29] TempDirectoryPath:: removing all files from /tmp/velox_test_H163pi
+```
+
+Here is the full list of query trace flags in MicroBenchmark.
+- query_trace_enabled: Whether to enable query trace.
+- query_trace_dir: Base dir of a query to store tracing data.
+- query_trace_node_ids: A comma-separated list of plan node ids whose input data will be traced. Empty string if only want to trace the query metadata.
+- query_trace_max_bytes: The max trace bytes limit. Tracing is disabled if zero.
+- query_trace_task_reg_exp: The regexp of traced task id. We only enable trace on a task if its id matches.
diff --git a/_docs/v1.5.0/developers/SubstraitModifications.md b/_docs/v1.5.0/developers/SubstraitModifications.md
new file mode 100644
index 0000000..6df5abf
--- /dev/null
+++ b/_docs/v1.5.0/developers/SubstraitModifications.md
@@ -0,0 +1,40 @@
+---
+layout: page
+title: SubstraitModifications
+nav_order: 8
+parent: Developers
+grand_parent: v1.5.0
+---
+# Substrait Modifications in Gluten
+
+Substrait is a project aiming to create a well-defined, cross-language specification for data compute operations.
+Since it is still under active development, there are some lacking representations for Gluten needed computing
+operations. At the same time, some existing representations need to be modified a bit to satisfy the needs of computing.
+
+
+In Gluten, the base version of Substrait is `v0.23.0`. This page records all the Gluten changes to Substrait proto
+files for reference. It is preferred to upstream these changes to Substrait, but for those cannot be upstreamed,
+alternatives like `AdvancedExtension` could be considered.
+
+## Modifications to algebra.proto
+
+* Added `JsonReadOptions` and `TextReadOptions` in `FileOrFiles`([#1584](https://github.com/apache/incubator-gluten/pull/1584)).
+* Changed join type `JOIN_TYPE_SEMI` to `JOIN_TYPE_LEFT_SEMI` and `JOIN_TYPE_RIGHT_SEMI`([#408](https://github.com/apache/incubator-gluten/pull/408)).
+* Added `WindowRel`, added `column_name` and `window_type` in `WindowFunction`,
+changed `Unbounded` in `WindowFunction` into `Unbounded_Preceding` and `Unbounded_Following`, and added WindowType([#485](https://github.com/apache/incubator-gluten/pull/485)).
+* Added `output_schema` in RelRoot([#1901](https://github.com/apache/incubator-gluten/pull/1901)).
+* Added `ExpandRel`([#1361](https://github.com/apache/incubator-gluten/pull/1361)).
+* Added `GenerateRel`([#574](https://github.com/apache/incubator-gluten/pull/574)).
+* Added `PartitionColumn` in `LocalFiles`([#2405](https://github.com/apache/incubator-gluten/pull/2405)).
+* Added `WriteRel` ([#3690](https://github.com/apache/incubator-gluten/pull/3690)).
+* Added `TopNRel` ([#5409](https://github.com/apache/incubator-gluten/pull/5409)).
+* Added `ref` field in window bound `Preceding` and `Following` ([#5626](https://github.com/apache/incubator-gluten/pull/5626)).
+* Added `BucketSpec` field in `WriteRel`([#8386](https://github.com/apache/incubator-gluten/pull/8386))
+* Added `StreamKafka` in `ReadRel`([#8321](https://github.com/apache/incubator-gluten/pull/8321))
+
+## Modifications to type.proto
+
+* Added `Nothing` in `Type`([#791](https://github.com/apache/incubator-gluten/pull/791)).
+* Added `names` in `Struct`([#1878](https://github.com/apache/incubator-gluten/pull/1878)).
+* Added `PartitionColumns` in `NamedStruct`([#320](https://github.com/apache/incubator-gluten/pull/320)).
+* Remove `PartitionColumns` and add `column_types` in `NamedStruct`([#2405](https://github.com/apache/incubator-gluten/pull/2405)).
diff --git a/_docs/v1.5.0/developers/UsingGperftoolsInCH.md b/_docs/v1.5.0/developers/UsingGperftoolsInCH.md
new file mode 100644
index 0000000..c142328
--- /dev/null
+++ b/_docs/v1.5.0/developers/UsingGperftoolsInCH.md
@@ -0,0 +1,33 @@
+---
+layout: page
+title: UsingGperftoolsInCH
+nav_order: 9
+parent: Developers
+grand_parent: v1.5.0
+---
+We need using gpertools to find the memory or CPU issue. That's what this document is about.
+
+## Install gperftools
+Install gperftools as described in https://github.com/gperftools/gperftools.
+We get the library and the command line tools.
+
+## Compiler libch.so
+Disable jemalloc `-DENABLE_JEMALLOC=OFF` in cpp-ch/CMakeLists.txt, and recompile libch.so.
+
+## Run Gluten with gperftools
+For Spark on Yarn, we can change the submit script to run Gluten with gperftools.
+Add the following to the submit script:
+```
+export tcmalloc_path=/data2/zzb/gperftools-2.10/.libs/libtcmalloc_and_profiler.so # the path to the tcmalloc library
+export LD_PRELOAD=$tcmalloc_path,libch.so # load the library in the driver
+--files $tcmalloc_path # upload the library to the cluster
+--conf spark.executorEnv.LD_PRELOAD=./libtcmalloc_and_profiler.so,libch.so # load the library in the executor
+--conf spark.executorEnv.HEAPPROFILE=/tmp/gluten_heap_perf # set the heap profile path, you can change to CPUPROFILE for CPU profiling
+```
+
+For thrift server on local machine, note using `export LD_PRELOAD="$tcmalloc_path libch.so" # load the library in the driver` to preload dynamic libraries.
+
+## Analyze the result
+We can get the result in the path we set in the previous step. For example, we can get the result in `/tmp/gluten_heap_perf`. We can use the following website to analyze the result:
+https://gperftools.github.io/gperftools/heapprofile.html
+https://gperftools.github.io/gperftools/cpuprofile.html
diff --git a/_docs/v1.5.0/developers/UsingJemallocWithCH.md b/_docs/v1.5.0/developers/UsingJemallocWithCH.md
new file mode 100644
index 0000000..6316560
--- /dev/null
+++ b/_docs/v1.5.0/developers/UsingJemallocWithCH.md
@@ -0,0 +1,47 @@
+---
+layout: page
+title: UsingJemallocWithCH
+nav_order: 10
+parent: Developers
+grand_parent: v1.5.0
+---
+We need using jemalloc to find the memory issue. That's what this document is about.
+
+## Change code of jemalloc
+Use libunwind instead of libgcc to get the backtrace which may cause process hangs.
+```
+@@ -177,7 +177,8 @@ target_compile_definitions(_jemalloc PRIVATE -DJEMALLOC_PROF=1)
+ # At the time ClickHouse uses LLVM libunwind which follows libgcc's way of backtracking.
+ #
+ # ClickHouse has to provide `unw_backtrace` method by the means of [commit 8e2b31e](https://github.com/ClickHouse/libunwind/commit/8e2b31e766dd502f6df74909e04a7dbdf5182eb1).
+-target_compile_definitions (_jemalloc PRIVATE -DJEMALLOC_PROF_LIBGCC=1)
++#target_compile_definitions (_jemalloc PRIVATE -DJEMALLOC_PROF_LIBGCC=1)
++target_compile_definitions (_jemalloc PRIVATE -DJEMALLOC_PROF_LIBUNWIND=1)
+ target_link_libraries (_jemalloc PRIVATE unwind)
+ ```
+For more infomation, check https://github.com/jemalloc/jemalloc/issues/2282.
+
+## Get jeprof
+Change to the directory where you want to install jemalloc, and run the following commands:
+```
+cd $Clickhouse_SOURCE_PATH/contrib/jemalloc && ./autogen.sh && ./configure.sh && make -j8
+```
+Then we get jeprof in the directory `$Clickhouse_SOURCE_PATH/contrib/jemalloc/bin/jeprof`.
+
+## Compiler libch.so
+Ensure to enable jemalloc `-DENABLE_JEMALLOC=ON` in cpp-ch/CMakeLists.txt, and compile libch.so.
+
+## Run Gluten with jemalloc heap tools
+For Yarn or thrift server, you need add the following to the submit script:
+```
+export MALLOC_CONF="prof:true,lg_prof_interval:30" # enable jemalloc heap profiling
+```
+You can find more options in https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling.
+
+## Analyze the result
+After you get the heap file, you can use the following command to analyze the result:
+Check memory diff, so you can find which part of the code consume the memory.
+```
+jeprof --svg /usr/lib/jvm/java-8-openjdk-amd64/bin/java --base=jeprof.3717358.1.i1.heap jeprof.3717358.30.i30.heap > diff.svg
+```
+You can find more usage about jeprof with `jeprof --help`.
\ No newline at end of file
diff --git a/_docs/v1.5.0/developers/VeloxDynamicSizingOffheap.md b/_docs/v1.5.0/developers/VeloxDynamicSizingOffheap.md
new file mode 100644
index 0000000..69fdc88
--- /dev/null
+++ b/_docs/v1.5.0/developers/VeloxDynamicSizingOffheap.md
@@ -0,0 +1,33 @@
+---
+layout: page
+title: VeloxDynamicSizingOffheap
+nav_order: 11
+parent: Developers
+grand_parent: v1.5.0
+---
+## Dynamic Off-heap Sizing
+Gluten requires setting both on-heap and off-heap memory sizes, which initializes different memory layouts. Improper configuration of these settings can lead to lower performance.
+
+To fix this issue, dynamic off-heap sizing is an experimental feature designed to simplify this process. When enabled, off-heap settings are ignored, and Velox uses the on-heap size as the memory size.
+## Detail implementations
+To understand the details, it's essential to learn the basics of JVM memory management. There are many resources discussing JVM internals:
+- https://exia.dev/blog/2019-12-10/JVM-Memory-Model/
+- https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/garbage_collect.html
+- https://www.scaler.com/topics/memory-management-in-java/
+- https://developers.redhat.com/articles/2021/09/09/how-jvm-uses-and-allocates-memory#
+
+In general, the feature works as follows:
+
+- Spark first attempts to allocate memory based on the on-heap size. Note that the maximum memory size is controlled by `spark.executor.memory`.
+- When Velox tries to allocate memory, Gluten attempts to allocate from system memory and records this in the memory allocator.
+- If there is sufficient memory, allocations proceed normally.
+- If memory is insufficient, Spark performs garbage collection (GC) to free on-heap memory, allowing Velox to allocate memory.
+- If memory remains insufficient after GC, Spark reports an out-of-memory (OOM) issue.
+
+We then enforce a total memory quota, calculated as the sum of committed and in-use memory in the Java heap (using `Runtime.getRuntime().totalMemory()`) plus tracked off-heap memory in `TreeMemoryConsumer`. If an allocation exceeds this total committed memory, the allocation fails and triggers an OOM.
+
+With this change, the "quota check" is performed when Gluten receives an memory allocation request. In practice, this means the Java codebase can oversubscribe memory within the on-heap quota, even if off-heap usage is sufficient to fail the allocation.
+
+## Limitations
+
+This feature is in the preliminary stages of development and will be improved in future updates.
\ No newline at end of file
diff --git a/_docs/v1.5.0/developers/VeloxUDF.md b/_docs/v1.5.0/developers/VeloxUDF.md
new file mode 100644
index 0000000..9101c6d
--- /dev/null
+++ b/_docs/v1.5.0/developers/VeloxUDF.md
@@ -0,0 +1,246 @@
+---
+layout: page
+title: VeloxUDF
+nav_order: 12
+parent: Developers
+grand_parent: v1.5.0
+---
+# Velox User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAF)
+
+## Introduction
+
+Velox backend supports User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAF).
+Users can implement custom functions using the UDF interface provided by Velox and compile them into libraries.
+At runtime, these UDFs are registered alongside their Java implementations via `CREATE TEMPORARY FUNCTION`.
+Once registered, Gluten can parse and offload these UDFs to Velox during execution,
+meanwhile ensuring proper fallback to Java UDFs when necessary.
+
+## Create and Build UDF/UDAF library
+
+The following steps demonstrate how to set up a UDF library project:
+
+- **Include the UDF Interface Header:**
+ First, include the UDF interface header file [Udf.h](../../cpp/velox/udf/Udf.h) in the project file.
+ The header file defines the `UdfEntry` struct, along with the macros for declaring the necessary functions to integrate the UDF into Gluten and Velox.
+
+- **Implement the UDF:**
+ Implement UDF. These functions should be able to register to Velox.
+
+- **Implement the Interface Functions:**
+ Implement the following interface functions that integrate UDF into Project Gluten:
+
+ - `getNumUdf()`:
+ This function should return the number of UDF in the library.
+ This is used to allocating udfEntries array as the argument for the next function `getUdfEntries`.
+
+ - `getUdfEntries(gluten::UdfEntry* udfEntries)`:
+ This function should populate the provided udfEntries array with the details of the UDF, including function names and signatures.
+
+ - `registerUdf()`:
+ This function is called to register the UDF to Velox function registry.
+ This is where users should register functions by calling `facebook::velox::exec::registerVectorFunction` or other Velox APIs.
+
+ - The interface functions are mapped to marcos in [Udf.h](../../cpp/velox/udf/Udf.h).
+
+ Assuming there is an existing Hive UDF `org.apache.gluten.sql.hive.MyUDF`, its native UDF can be implemented as follows.
+
+ ```
+ #include <velox/expression/VectorFunction.h>
+ #include <velox/udf/Udf.h>
+
+ namespace {
+ static const char* kInteger = "integer";
+ static const char* kMyUdfFunctionName = "org.apache.gluten.sql.hive.MyUDF";
+ }
+
+ const int kNumMyUdf = 1;
+
+ const char* myUdfArgs[] = {kInteger}:
+ gluten::UdfEntry myUdfSig = {kMyUdfFunctionName, kInteger, 1, myUdfArgs};
+
+ class MyUdf : public facebook::velox::exec::VectorFunction {
+ ... // Omit concrete implementation
+ }
+
+ static std::vector<std::shared_ptr<exec::FunctionSignature>>
+ myUdfSignatures() {
+ return {facebook::velox::exec::FunctionSignatureBuilder()
+ .returnType(myUdfSig.dataType)
+ .argumentType(myUdfSig.argTypes[0])
+ .build()};
+ }
+
+ DEFINE_GET_NUM_UDF { return kNumMyUdf; }
+
+ DEFINE_GET_UDF_ENTRIES { udfEntries[0] = myUdfSig; }
+
+ DEFINE_REGISTER_UDF {
+ facebook::velox::exec::registerVectorFunction(
+ myUdf[0].name, myUdfSignatures(), std::make_unique<MyUdf>());
+ }
+
+ ```
+
+To build the UDF library, users need to compile the C++ code and link to `libvelox.so`.
+It's recommended to create a CMakeLists.txt for the project. Here's an example:
+
+```
+project(myudf)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+
+set(GLUTEN_HOME /path/to/gluten)
+
+add_library(myudf SHARED "MyUDF.cpp")
+
+find_library(VELOX_LIBRARY REQUIRED NAMES velox HINTS ${GLUTEN_HOME}/cpp/build/releases NO_DEFAULT_PATH)
+
+target_include_directories(myudf PRIVATE ${GLUTEN_HOME}/cpp ${GLUTEN_HOME}/ep/build-velox/build/velox_ep)
+target_link_libraries(myudf PRIVATE ${VELOX_LIBRARY})
+```
+
+The steps for creating and building a UDAF library are quite similar to those for a UDF library.
+The major difference lies in including and defining specific functions within the UDAF header file [Udaf.h](../../cpp/velox/udf/Udaf.h)
+
+- `getNumUdaf()`
+- `getUdafEntries(gluten::UdafEntry* udafEntries)`
+- `registerUdaf()`
+
+`gluten::UdafEntry` requires an additional field `intermediateType`, to specify the output type from partial aggregation.
+For detailed implementation, you can refer to the example code in [MyUDAF.cc](../../cpp/velox/udf/examples/MyUDAF.cc)
+
+## Using UDF/UDAF in Gluten
+
+Gluten loads the UDF libraries at runtime. You can upload UDF libraries via `--files` or `--archives`, and configure the library paths using the provided Spark configuration, which accepts comma separated list of library paths.
+
+Note if running on Yarn client mode, the uploaded files are not reachable on driver side. Users should copy those files to somewhere reachable for driver and set `spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths`.
+This configuration is also useful when the `udfLibraryPaths` is different between driver side and executor side.
+
+- Use the `--files` option to upload a library and configure its relative path
+
+```shell
+--files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
+# Needed for Yarn client mode
+--conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+```
+
+- Use the `--archives` option to upload an archive and configure its relative path
+
+```shell
+--archives /path/to/udf_archives.zip#udf_archives
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=udf_archives
+# Needed for Yarn client mode
+--conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/udf_archives.zip
+```
+
+- Configure URI
+
+You can also specify the local or HDFS URIs to the UDF libraries or archives. Local URIs should exist on driver and every worker nodes.
+
+```shell
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///path/to/library_or_archive
+```
+
+## Try the example
+
+We provided Velox UDF examples in file [MyUDF.cc](../../cpp/velox/udf/examples/MyUDF.cc) and UDAF examples in file [MyUDAF.cc](../../cpp/velox/udf/examples/MyUDAF.cc).
+You need to build the gluten project with `--build_example=ON` to get the example libraries.
+
+```shell
+./dev/buildbundle-veloxbe.sh --build_examples=ON
+```
+
+Then, you can find the example libraries at /path/to/gluten/cpp/build/velox/udf/examples/
+
+Start spark-shell or spark-sql with below configuration
+
+```shell
+# Use the `--files` option to upload a library and configure its relative path
+--files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
+```
+
+or
+
+```shell
+# Only configure URI
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=file:///path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+```
+
+Start `spark-sql` and run query. You need to add jar "spark-hive_2.12-<spark.version>-tests.jar" to the classpath for hive udf `org.apache.spark.sql.hive.execution.UDFStringString`
+
+```
+spark-sql (default)> create table tbl as select * from values ('hello');
+Time taken: 3.656 seconds
+spark-sql (default)> CREATE TEMPORARY FUNCTION hive_string_string AS 'org.apache.spark.sql.hive.execution.UDFStringString';
+Time taken: 0.047 seconds
+spark-sql (default)> select hive_string_string(col1, 'world') from tbl;
+hello world
+Time taken: 1.217 seconds, Fetched 1 row(s)
+```
+
+You can verify the offload with "explain".
+```
+spark-sql (default)> explain select hive_string_string(col1, 'world') from tbl;
+VeloxColumnarToRow
++- ^(2) ProjectExecTransformer [HiveSimpleUDF#org.apache.spark.sql.hive.execution.UDFStringString(col1#11,world) AS hive_string_string(col1, world)#12]
+ +- ^(2) InputIteratorTransformer[col1#11]
+ +- RowToVeloxColumnar
+ +- Scan hive spark_catalog.default.tbl [col1#11], HiveTableRelation [`spark_catalog`.`default`.`tbl`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [col1#11], Partition Cols: []]
+```
+
+## Configurations
+
+| Parameters | Description |
+|----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
+| spark.gluten.sql.columnar.backend.velox.udfLibraryPaths | Path to the udf/udaf libraries. |
+| spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths | Path to the udf/udaf libraries on driver node. Only applicable on yarn-client mode. |
+| spark.gluten.sql.columnar.backend.velox.udfAllowTypeConversion | Whether to inject possible `cast` to convert mismatched data types from input to one registered signatures. |
+
+# Pandas UDFs (a.k.a. Vectorized UDFs)
+
+## Introduction
+
+Pandas UDFs are user defined functions that are executed by Spark using Arrow to transfer data and Pandas to work with the data, which allows vectorized operations. A Pandas UDF is defined using the pandas_udf() as a decorator or to wrap the function, and no additional configuration is required.
+A Pandas UDF behaves as a regular PySpark function API in general. For more details, you can refer [doc](https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html).
+
+## Using Pandas UDFs in Gluten with Velox Backend
+
+Similar as in vanilla Spark, user needs to set up pyspark/arrow dependencies properly first. You may can refer following steps:
+
+```
+pip3 install pyspark==$SPARK_VERSION cython
+pip3 install pandas pyarrow
+```
+
+Gluten provides a config to control enable `ColumnarArrowEvalPython` or not, with `true` as default.
+
+```
+spark.gluten.sql.columnar.arrowUdf
+```
+
+Then take following `PySpark` code for example:
+
+```
+from pyspark.sql.functions import pandas_udf, PandasUDFType
+import pyspark.sql.functions as F
+import os
+@pandas_udf('long')
+def pandas_plus_one(v):
+ return (v + 1)
+df = spark.read.orc("path_to_file").select("quantity").withColumn("processed_quantity", pandas_plus_one("quantity")).select("quantity")
+```
+
+The expected physical plan will be:
+
+```
+== Physical Plan ==
+VeloxColumnarToRowExec
++- ^(2) ProjectExecTransformer [pythonUDF0#45L AS processed_quantity#41L]
+ +- ^(2) InputIteratorTransformer[quantity#2L, pythonUDF0#45L]
+ +- ^(2) InputAdapter
+ +- ^(2) ColumnarArrowEvalPython [pandas_plus_one(quantity#2L)#40L], [pythonUDF0#45L], 200
+ +- ^(1) NativeFileScan orc [quantity#2L] Batched: true, DataFilters: [], Format: ORC, Location: InMemoryFileIndex(1 paths)[file:/***], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<quantity:bigint>
+```
diff --git a/_docs/v1.5.0/developers/clickhouse-backend-debug.md b/_docs/v1.5.0/developers/clickhouse-backend-debug.md
new file mode 100644
index 0000000..3482a2f
--- /dev/null
+++ b/_docs/v1.5.0/developers/clickhouse-backend-debug.md
@@ -0,0 +1,97 @@
+---
+layout: page
+title: clickhouse backend debug
+nav_order: 13
+parent: Developers
+grand_parent: v1.5.0
+---
+# Debug CH Backend
+
+## Debug Java/Scala Code with IntelliJ IDEA
+
+1. Build Gluten ClickHouse Native Lib.
+ ```
+ export CMAKE_BUILD_TYPE=Release && bash ep/build-clickhouse/src/build_clickhouse.sh
+ ```
+ libch.so will be generated in `cpp-ch/build/utils/extern-local-engine/libch.so`.
+
+2. Maven Build Gluten ClickHouse with Profile
+ ```
+ mvn clean install -DskipTests -P delta -Pbackends-clickhouse -Pspark-3.3 -Pspark-ut
+ ```
+
+3. Set Maven Profiles in IntelliJ IDEA
+
+ 
+
+4. Set Debug Configuration in IntelliJ IDEA
+
+ For example, debug GlutenMathExpressionsSuite.
+ 
+
+ VM Options:
+ `-Dgluten.test.data.path=/data -Dspark.gluten.sql.columnar.libpath=/path/to/gluten/cpp-ch/build/utils/extern-local-engine/libch.so -Dspark.test.home=/path/to/spark33`
+ > Download tpcds-data in https://gluten-nginx.kyligence.com/dataset/
+ > Download spark33 using `git clone --depth 1 --branch v3.3.1 https://github.com/apache/spark.git /tmp/spark33`
+
+ Environment Variables:
+ `LD_PRELOAD=/path/to/gluten/cpp-ch/build/utils/extern-local-engine/libch.so:/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/libjsig.so`
+
+## Debug Native Code with CLion
+
+1. Toolchains Settings
+
+ 
+ > Some Clion versions have not supported lldb-18, you can manually set refer to [CLion Custom LLDB](https://youtrack.jetbrains.com/issue/CPP-3589/Support-using-custom-LLDB). Or you can use gdb as default.
+
+2. CMake Debug Configuration
+
+ 
+
+ CMake Options:
+ `-DENABLE_PROTOBUF=ON -DENABLE_TESTS=OFF -DENABLE_BENCHMARKS=ON -DENABLE_JEMALLOC=ON -DENABLE_MULTITARGET_CODE=ON -DENABLE_EXTERN_LOCAL_ENGINE=ON -DENABLE_ODBC=OFF -DENABLE_CAPNP=OFF -DENABLE_ROCKSDB=OFF -DENABLE_GRPC=OFF -DENABLE_RUST=OFF -DENABLE_H3=OFF -DENABLE_AMQPCPP=OFF -DENABLE_CASSANDRA=OFF -DENABLE_KAFKA=ON -DENABLE_NATS=OFF -DENABLE_LIBPQXX=OFF -DENABLE_NURAFT=OFF -DENABLE_DATASKETCHES=OFF -DENABLE_SQLITE=OFF -DENABLE_S2_GEOMETRY=OFF -DENABLE_ANNOY=OFF -DENABLE_ULID=OFF -DEN [...]
+
+3. Reload CMake Project
+
+ 
+
+ After reload cmake projects, you can find target `libch` in run configurations.
+
+4. Build `libchd.so` with Debug Mode
+
+ 
+
+ `libchd.so` will be generated in `cmake-build-debug/utils/extern-local-engine/libchd.so`.
+
+5. Create File `.gdbinit` and `.lldbinit` to Avoid Unused Signal
+
+ vi ~/.gdbinit
+ ```
+ handle SIGSEGV nostop noprint
+
+ set print pretty on
+ set print object on
+ python
+ import sys
+ sys.path.insert(0, '/path/to/libcxx-pretty-printers/src')
+ from libcxx.v1.printers import register_libcxx_printers
+ register_libcxx_printers(None)
+ end
+ ```
+ > Download libcxx-pretty-printers in https://github.com/koutheir/libcxx-pretty-printers
+
+ vi ~/.lldbinit
+ ```
+ process handle -n true -p true -s false SIGBUS SIGSEGV
+ ```
+
+6. Debug Application
+
+ 
+
+ Executable: `/path/to/java`
+
+ Program Arguments: Copy from IntelliJ IDEA debug command line and remove `-javaagent`
+ 
+
+
diff --git a/_docs/v1.5.0/developers/improvement-proposals.md b/_docs/v1.5.0/developers/improvement-proposals.md
new file mode 100644
index 0000000..594caf7
--- /dev/null
+++ b/_docs/v1.5.0/developers/improvement-proposals.md
@@ -0,0 +1,100 @@
+---
+layout: page
+title: improvement proposals
+nav_order: 14
+parent: Developers
+grand_parent: v1.5.0
+---
+# Gluten Project Improvement Proposals (GPIP)
+
+The Gluten Project Improvement Proposals doc references [the Spark SPIP documentation](https://spark.apache.org/improvement-proposals.html).
+
+The purpose of a GPIP is to inform and involve the user community in major improvements to the Gluten codebase throughout the development process to increase the likelihood that user needs are met.
+
+GPIPs should be used for significant user-facing or cutting-edge changes, not small incremental improvements.
+
+If your proposal meets the definition of GPIP, we recommend you to create a GPIP, which will facilitate the advancement and discussion of the proposal, but it is not mandatory, and we welcome any contribution and community participation.
+
+## What is a GPIP?
+
+A GPIP is similar to a product requirement document commonly used in product management.
+
+A GPIP:
+
+- Is a ticket labeled “GPIP” proposing a major improvement or change to Gluten
+- Follows the template defined below
+- Includes discussions on the ticket and dev@ list about the proposal
+
+## Who?
+
+Any **community member** can help by discussing whether a GPIP is likely to meet their needs and propose GPIPs.
+
+**Contributors** can help by discussing whether a GPIP is likely to be technically feasible.
+
+**Committers** can help by discussing whether a GPIP aligns with long-term project goals, and by shepherding GPIPs.
+
+**GPIP Author** is any community member who authors a GPIP and is committed to pushing the change through the entire process. GPIP authorship can be transferred.
+
+**GPIP Shepherd** is a PMC member who is committed to shepherding the proposed change throughout the entire process. Although the shepherd can delegate or work with other committers in the development process, the shepherd is ultimately responsible for the success or failure of the GPIP. Responsibilities of the shepherd include, but are not limited to:
+
+- Be the advocate for the proposed change
+- Help push forward on design and achieve consensus among key stakeholders
+- Review code changes, making sure the change follows project standards
+- Get feedback from users and iterate on the design & implementation
+- Uphold the quality of the changes, including verifying whether the changes satisfy the goal of the GPIP and are absent of critical bugs before releasing them
+
+## GPIP Process
+### Proposing a GPIP
+
+Anyone may propose a GPIP, using the document template below. Please only submit a GPIP if you are willing to help, at least with discussion.
+
+After a GPIP is created, the author should email dev@gluten.apache.org to notify the community of the GPIP, and discussions should ensue on the ticket.
+
+If a GPIP is too small or incremental and should have been done through the normal ticket process, a committer should remove the GPIP label.
+
+### GPIP Document Template
+
+A GPIP document is a short document with a few questions, inspired by the Heilmeier Catechism:
+
+- Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.
+
+- Q2. What problem is this proposal NOT designed to solve?
+
+- Q3. How is it done today, and what are the limits of current practice?
+
+- Q4. What is new in your approach, and why do you think it will be successful?
+
+- Q5. Who cares? If you are successful, what difference will it make?
+
+- Q6. What are the risks?
+
+- Q7. How long will it take?
+
+- Q8. What are the mid-term and final “exams” to check for success?
+
+- Appendix A. Proposed API Changes. Optional section defining APIs changes, if any. Backward and forward compatibility must be taken into account.
+
+- Appendix B. Optional Design Sketch: How are the goals going to be accomplished? Give sufficient technical detail to allow a contributor to judge whether it's likely to be feasible. Note that this is not a full design document.
+
+- Appendix C. Optional Rejected Designs: What alternatives were considered? Why were they rejected? If no alternatives have been considered, the problem needs more thought.
+
+### Discussing a GPIP
+
+All discussions of a GPIP should take place in a public forum, preferably the discussion attached to the ticket. Any discussion that happen offline should be made available online for the public via meeting notes summarizing the discussions.
+
+During this discussion, one or more shepherds should be identified among PMC members.
+
+Once the discussion settles, the shepherd(s) should call for a vote on the GPIP moving forward on the dev@ list. The vote should be open for at least 72 hours and follows the typical Apache vote process and passes upon consensus (at least 3 +1 votes from PMC members and no -1 votes from PMC members). dev@ should be notified of the vote result.
+
+If there does not exist at least one PMC member that is committed to shepherding the change within a month, the GPIP is rejected.
+
+If a committer does not think a GPIP aligns with long-term project goals, or is not practical at the point of proposal, the committer should -1 the GPIP explicitly and give technical justifications.
+
+
+### Implementing a GPIP
+
+Implementation should take place via the contribution guidelines. Changes that require GPIPs typically also require design documents to be written and reviewed.
+
+### GPIP Community collaboration
+
+The gluten community has always been open to contributions and participation of all kinds, and feel free to contact the community if you have any questions about the GPIP process.
diff --git a/_docs/v1.5.0/developers/index.md b/_docs/v1.5.0/developers/index.md
new file mode 100644
index 0000000..915983a
--- /dev/null
+++ b/_docs/v1.5.0/developers/index.md
@@ -0,0 +1,11 @@
+---
+layout: page
+title: Developers
+nav_order: 3
+has_children: true
+parent: v1.5.0
+permalink: /archives/v1.5.0/developers
+---
+# Gluten Developers
+This document provides a developer overview of the project and covers the
+following topics:
diff --git a/_docs/v1.5.0/developers/velox-backend-CI.md b/_docs/v1.5.0/developers/velox-backend-CI.md
new file mode 100644
index 0000000..eb597c8
--- /dev/null
+++ b/_docs/v1.5.0/developers/velox-backend-CI.md
@@ -0,0 +1,42 @@
+---
+layout: page
+title: velox backend CI
+nav_order: 15
+parent: Developers
+grand_parent: v1.5.0
+---
+# Velox Backend CI
+
+GHA workflows are defined under `.github/workflows/`.
+
+## Docker Build
+We have a weekly job defined in `docker_image.yml` to build docker images for CI verification. The docker files and images are listed below:
+
+file | images | comments
+-- | -- | --
+dev/docker/Dockerfile.centos7-static-build | apache/gluten:vcpkg-centos-7 | centos 7, static link, jdk8
+dev/docker/Dockerfile.centos8-static-build | apache/gluten:vcpkg-centos-8 | centos 8, static link, jdk8
+dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk8 | centos 8, dynamic link, jdk8
+dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk11 | centos 8, dynamic link, jdk11
+dev/docker/Dockerfile.centos8-dynamic-build | apache/gluten:centos-8-jdk17 | centos 8, dynamic link, jdk17
+dev/docker/cudf/Dockerfile | apache/gluten:centos-9-jdk8-cudf | centos 9, dynamic link, jdk8
+
+Docker images can be found from https://hub.docker.com/r/apache/gluten/tags
+
+## Vcpkg Caching
+Gluten main branch is pulled down during static build in docker. And vcpkg will cache binary data of all dependencies defined under dev/vcpkg.
+These binary data is cached into `/var/cache/vcpkg` and CI job can re-use them in new build. By setting `VCPKG_BINARY_SOURCES=clear` in env.,
+reusing vcpkg cache can be disabled.
+
+## Arrow Libs Pre-installation
+Arrow libs are pre-installed in docker, assuming they are not actively changed, then not necessarily to be re-built every time.
+
+## .M2 cache
+The dependency libraries are pre installed in to /root/.m2 by `mvn dependency:go-offline` Spark is set to 3.5 by default.
+
+## Ccache
+Since the docker image is rebuilt weekly, the ccache mostly are outdated. So the cache is removed from the image.
+
+## Updating Docker Image
+Two GitHub secrets `DOCKERHUB_USER` & `DOCKERHUB_TOKEN` can be used to push docker image to docker hub: https://hub.docker.com/r/apache/gluten/tags.
+Note GitHub secrets are not retrievable in PR from forked repo.
\ No newline at end of file
diff --git a/_docs/v1.5.0/developers/velox-backend-build-in-docker.md b/_docs/v1.5.0/developers/velox-backend-build-in-docker.md
new file mode 100644
index 0000000..87ce340
--- /dev/null
+++ b/_docs/v1.5.0/developers/velox-backend-build-in-docker.md
@@ -0,0 +1,68 @@
+---
+layout: page
+title: velox backend build in docker
+nav_order: 16
+parent: Developers
+grand_parent: v1.5.0
+---
+Currently, we have two way to build Gluten, static link or dynamic link.
+
+# Static link
+The static link approach builds all dependency libraries in vcpkg for both Velox and Gluten. It then statically links these libraries into libvelox.so and libgluten.so, enabling the build of Gluten on *any* Linux OS on x86 platforms with 64G memory. However we only verified on Centos-7/8/9 and Ubuntu 20.04/22.04. Please submit an issue if it fails on your OS.
+
+Here is the dependency libraries required on target system, they are the essential libraries pre-installed in every Linux OS.
+```
+linux-vdso.so.1
+librt.so.1
+libpthread.so.0
+libdl.so.2
+libm.so.6
+libc.so.6
+/lib64/ld-linux-x86-64.so.2
+```
+
+The 'dockerfile' to build Gluten jar:
+
+```
+FROM apache/gluten:vcpkg-centos-7
+
+# Build Gluten Jar
+RUN source /opt/rh/devtoolset-11/enable && \
+ git clone https://github.com/apache/incubator-gluten.git && \
+ cd incubator-gluten && \
+ ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_s3=ON --enable_gcs=ON --enable_abfs=ON --enable_vcpkg=ON --build_arrow=OFF && \
+ mvn clean package -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Pspark-3.4 -DskipTests
+```
+`enable_vcpkg=ON` enables the static link. Vcpkg packages are already pre-installed in the vcpkg-centos-7 image and can be reused automatically. The image is maintained by Gluten community.
+
+The command builds Gluten jar in 'glutenimage':
+```
+docker build -t glutenimage -f dockerfile
+```
+The gluten jar can be copied from glutenimage:/incubator-gluten/package/target/gluten-velox-bundle-*.jar
+
+# Dynamic link
+The dynamic link approach needs to install the dependencies libraries. It then dynamically link the .so files into libvelox.so and libgluten.so. Currently, Centos-7/8/9 and
+ Ubuntu 20.04/22.04 are supported to build Gluten Velox backend dynamically.
+
+The 'dockerfile' to build Gluten jar:
+
+```
+FROM apache/gluten:centos-8-jdk8
+
+# Build Gluten Jar
+RUN source /opt/rh/devtoolset-11/enable && \
+ git clone https://github.com/apache/incubator-gluten.git && \
+ cd incubator-gluten && \
+ ./dev/builddeps-veloxbe.sh --run_setup_script=ON --enable_hdfs=ON --enable_vcpkg=OFF --build_arrow=OFF && \
+ mvn clean package -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Pspark-3.4 -DskipTests && \
+ ./dev/build-thirdparty.sh
+```
+`enable_vcpkg=OFF` enables the dynamic link. Part of shared libraries are pre-installed in the image. You need to specify `--run_setup_script=ON` to install the rest of them. It then packages all dependency libraries into a jar by `build-thirdparty.sh`.
+Please note the image is built based on centos-8. It has risk to build and deploy the jar on other OSes.
+
+The command builds Gluten jar in 'glutenimage':
+```
+docker build -t glutenimage -f dockerfile
+```
+The gluten jar can be copied from glutenimage:/incubator-gluten/package/target/gluten-velox-bundle-*.jar and glutenimage:/incubator-gluten/package/target/gluten-thirdparty-lib-*.jar
diff --git a/_docs/v1.5.0/developers/velox-function-development-guide.md b/_docs/v1.5.0/developers/velox-function-development-guide.md
new file mode 100644
index 0000000..d172210
--- /dev/null
+++ b/_docs/v1.5.0/developers/velox-function-development-guide.md
@@ -0,0 +1,46 @@
+---
+layout: page
+title: velox function development guide
+nav_order: 17
+parent: Developers
+grand_parent: v1.5.0
+---
+# Developer Guide for Implementing Spark Built-in SQL Functions in Velox
+
+In velox, two folders `prestosql` & `sparksql` are holding most sql functions, respective for `presto` and `spark`. Gluten will ask velox to firstly register `prestosql` functions, then `sparksql` functions. So if `prestosql`
+and `sparksql` share same signature for a function, the `sparksql` function will overwrite the corresponding `prestosql` function. If the required function is lacking in both folders (exceptions are some common functions defined
+outside, like `cast`), we need to implement the missing function in `sparksql` folder. It is possible that a `prestosql` function has some semantic difference with the corresponding spark function, even though they share the
+same name and function signature. If so, we also need to do an implementation in `sparksql` folder, generally based on the original impl. for `prestosql`.
+
+There are a few spark functions that can behave differently for some special cases, depending on ANSI on or off. Currently, gluten does NOT support ANSI mode. So only ANSI off needs to be considered in implementing spark
+built-in functions in velox.
+
+Take `BitwiseAndFunction` as example:
+
+```
+template <typename T>
+struct BitwiseAndFunction {
+ template <typename TInput>
+ // For void return type, it indicates null result will never be obtained for non-null input.
+ // For bool return type, it indicates null result can be obtained for non-null input (false for null).
+ FOLLY_ALWAYS_INLINE void call(TInput& result, TInput a, TInput b) {
+ result = a & b;
+ }
+};
+```
+It is templated, as well as the `call` function, to allow multiple types. In the above impl., the result will be null for null input.
+Please use `callNullable` if you need different behavior for null input, e.g., get a non-null result for null input. Also see `callNullFree` in velox document.
+It is used for fast evaluation in the case that any input has null.
+
+The below code will register the implemented function for all kinds of integer types. The specified name `bitwise_and` will be actually used in calling this function.
+```
+registerBinaryIntegral<BitwiseAndFunction>({prefix + "bitwise_and"});
+```
+
+Functions for complex types have similar implementations.
+See `ArrayAverageFunction` in [velox/functions/prestosql/ArrayFunctions.h](https://github.com/facebookincubator/velox/blob/main/velox/functions/prestosql/ArrayFunctions.h).
+
+### Reference:
+Velox’s official developer guide:
+ * [velox/docs/develop/scalar-functions.rst](https://github.com/facebookincubator/velox/blob/main/velox/docs/develop/scalar-functions.rst)
+ * [velox/examples/SimpleFunctions.cpp](https://github.com/facebookincubator/velox/blob/main/velox/examples/SimpleFunctions.cpp)
diff --git a/_docs/v1.5.0/get-started/ClickHouse.md b/_docs/v1.5.0/get-started/ClickHouse.md
new file mode 100644
index 0000000..4c68b4f
--- /dev/null
+++ b/_docs/v1.5.0/get-started/ClickHouse.md
@@ -0,0 +1,727 @@
+---
+layout: page
+title: ClickHouse
+nav_order: 1
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+## ClickHouse Backend
+
+ClickHouse is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP), which supports best in the industry query performance, while significantly reducing storage requirements through its innovative use of columnar storage and compression.
+We port ClickHouse ( based on version **23.1** ) as a library, called 'libch.so', and Gluten loads this library through JNI as the native engine. In this way, we don't need to deploy a standalone ClickHouse Cluster, Spark uses Gluten as SparkPlugin to read and write ClickHouse MergeTree data.
+
+### Architecture
+
+The architecture of the ClickHouse backend is shown below:
+
+
+
+1. On Spark driver, Spark uses Gluten SparkPlugin to transform the physical plan to the Substrait plan, and then pass the Substrait plan to ClickHouse backend through JNI call on executors.
+2. Based on Spark DataSource V2 interface, implementing a ClickHouse Catalog to support operating the ClickHouse tables, and then using Delta to save some metadata about ClickHouse like the MergeTree parts information, and also provide ACID transactions.
+3. When querying from a ClickHouse table, it will fetch MergeTree parts information from Delta metadata and assign these parts into Spark partitions according to some strategies.
+4. When writing data into a ClickHouse table, it will use ClickHouse library to write MergeTree parts data and collect these MergeTree parts information after writing successfully, and then save these MergeTree parts information into Delta metadata. ( **The feature of writing MergeTree parts is coming soon.** )
+5. On Spark executors, each executor will load the 'libch.so' through JNI when starting, and then call the operators according to the Substrait plan which is passed from Spark Driver, like reading data from the MergeTree parts, writing the MergeTree parts, filtering data, aggregating data and so on.
+6. Currently, the ClickHouse backend only supports reading the MergeTree parts from local storage, it needs to use a high-performance shared file system to share a root bucket on every node of the cluster from the object storage, like JuiceFS.
+
+
+### Development environment setup
+
+In general, we use IDEA for Gluten development and CLion for ClickHouse backend development on **Ubuntu 20**.
+
+#### Prerequisite
+
+##### For compiling clickhouse backend
+
+Following softwares are required,
+- Clang 19.0
+- cmake 3.20 or higher version
+- ninja-build 1.8.2
+
+You can run `sudo $gluten_root/ep/build-clickhouse/src/install_ubuntu.sh` to setup the requirements. We also provide a [docker file](../../cpp-ch/local-engine/docker/image/Dockerfile), you can build your own image
+```shell
+cd $gluten_root/cpp-ch/local-engine/docker/image/
+docker build . -t libch_builder
+```
+
+You can also refer to [How-to-Build-ClickHouse-on-Linux](https://clickhouse.com/docs/en/development/build/).
+
+##### For compiling gluten
+You need to install the following software manually:
+- Java 8
+- Maven 3.6.3 or higher version
+- Spark 3.2.2 or Spark 3.3.1
+
+Then, get Gluten code:
+```shell
+git clone https://github.com/apache/incubator-gluten.git
+```
+
+#### Setup ClickHouse backend development environment
+
+##### Compile Clickhouse backend
+
+###### clone repos
+clone gluten repo
+
+
+```shell
+git clone https://github.com/apache/incubator-gluten.git
+```
+
+clone Kyligence/ClickHouse repo
+
+```shell
+git clone --recursive --shallow-submodules -b clickhouse_backend https://github.com/Kyligence/ClickHouse.git
+```
+
+checkout to the latest branch
+
+```shell
+latest_branch=$(cat $gluten_root/cpp-ch/clickhouse.version | grep CH_BRANCH | cut -d= -f2)
+git fetch origin $latest_branch
+git checkout -b $latest_branch origin/$latest_branch
+git submodule sync --recursive
+git submodule update --init --recursive
+```
+
+
+
+
+##### build
+
+There are several ways to build the backend library.
+1. Build it direclty
+
+
+If you have setup all requirements, you can use following command to build it direclty.
+
+```bash
+cd $gluten_root
+bash ./ep/build-clickhouse/src/build_clickhouse.sh
+```
+
+
+This will download Clickhouse for you and build everything.
+The target file is `$gluten_root/cpp-ch/build/utils/extern-local-engine/libch.so`.
+
+
+2. Use docker
+
+
+You can use [docker file](../../cpp-ch/local-engine/docker/image/Dockerfile) to build a docker image
+```shell
+cd $gluten_root/cpp-ch/local-engine/docker/image/
+docker build . -t libch_builder
+
+cd $gluten_root/cpp-ch/local-engine/docker
+./build.sh -g <gluten_root> -c <clickhouse_root> [-b <build_directory>] [-o <output_directory>] libch_builder
+```
+`build_directory` is a directory used as to store the intermediate files from compiling. It will use `current_dir>`/build as the default value if you don't provide it.
+
+
+`output_directory` is used as to store the finally output `libch.so`. The default value is `current_dir`/output if you don't provide it.
+
+
+3. (Option 1) Use CLion
+
+ - Open ClickHouse repo
+ - Choose File -> Settings -> Build, Execution, Deployment -> Toolchains, and then choose Bundled CMake, clang-16 as C Compiler, clang++-16 as C++ Compiler:
+
+ 
+
+ - Choose File -> Settings -> Build, Execution, Deployment -> CMake:
+
+ 
+
+ And then add these options into CMake options:
+ ```
+ -DENABLE_PROTOBUF=ON -DENABLE_TESTS=OFF -DENABLE_JEMALLOC=ON -DENABLE_MULTITARGET_CODE=ON -DENABLE_EXTERN_LOCAL_ENGINE=ON
+ ```
+ - Build 'ch' target on ClickHouse Project with Debug mode or Release mode:
+
+ 
+
+ If it builds with Release mode successfully, there is a library file called 'libch.so' in path '${CH_SOURCE_DIR}/cmake-build-release/utils/extern-local-engine/'.
+
+ If it builds with Debug mode successfully, there is a library file called 'libchd.so' in path '${CH_SOURCE_DIR}/cmake-build-debug/utils/extern-local-engine/'.
+
+4. Build it as a submodule of Clickhouse
+
+
+```shell
+ln -s $gluten_root/cpp-ch/local-engine $clickhouse_root/utils/extern-local-engine
+mkdir -p $clickhouse_root/build
+cd $clickhouse_root/build
+cmake -G Ninja "-DCMAKE_C_COMPILER=$CC" "-DCMAKE_CXX_COMPILER=$CXX" \
+ "-DCMAKE_BUILD_TYPE=Release" \
+ "-DENABLE_PROTOBUF=1" \
+ "-DENABLE_EMBEDDED_COMPILER=$ENABLE_EMBEDDED_COMPILER" \
+ "-DENABLE_TESTS=OFF" \
+ "-DWERROR=OFF" \
+ "-DENABLE_JEMALLOC=1" \
+ "-DENABLE_MULTITARGET_CODE=ON" \
+ "-DENABLE_GWP_ASAN=OFF" \
+ "-DENABLE_EXTERN_LOCAL_ENGINE=ON" \
+ "-DENABLE_THINLTO=false" \
+ ..
+ninja
+```
+The result is in `$clickhouse_root/build/utils/extern-local-engine/libch.so`.
+
+
+
+#### Compile Gluten
+
+The prerequisites are the same as the one mentioned above. Compile Gluten with ClickHouse backend through maven:
+
+- for Spark 3.2.2<span id="deploy-spark-322"></span>
+
+```
+ git clone https://github.com/apache/incubator-gluten.git
+ cd incubator-gluten/
+ export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
+ mvn clean install -Pbackends-clickhouse -Phadoop-2.7.4 -Pspark-3.2 -Dhadoop.version=2.8.5 -DskipTests -Dcheckstyle.skip
+ ls -al backends-clickhouse/target/gluten-XXXXX-spark-3.2-jar-with-dependencies.jar
+```
+
+- for Spark 3.3.1
+
+```
+ git clone https://github.com/apache/incubator-gluten.git
+ cd incubator-gluten/
+ export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
+ mvn clean install -Pbackends-clickhouse -Phadoop-2.7.4 -Pspark-3.3 -Dhadoop.version=2.8.5 -DskipTests -Dcheckstyle.skip
+ ls -al backends-clickhouse/target/gluten-XXXXX-spark-3.3-jar-with-dependencies.jar
+```
+
+### Gluten in local Spark Thrift Server
+
+#### Prepare working directory
+
+- for Spark 3.2.2
+
+```
+tar zxf spark-3.2.2-bin-hadoop2.7.tgz
+cd spark-3.2.2-bin-hadoop2.7
+#download delta-core_2.12-2.0.1.jar and delta-storage-2.0.1.jar
+wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.0.1/delta-core_2.12-2.0.1.jar -P ./jars
+wget https://repo1.maven.org/maven2/io/delta/delta-storage/2.0.1/delta-storage-2.0.1.jar -P ./jars
+cp gluten-XXXXX-spark-3.2-jar-with-dependencies.jar jars/
+```
+
+- for Spark 3.3.1
+
+```
+tar zxf spark-3.3.1-bin-hadoop2.7.tgz
+cd spark-3.3.1-bin-hadoop2.7
+#download delta-core_2.12-2.2.0.jar and delta-storage-2.2.0.jar
+wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.2.0/delta-core_2.12-2.2.0.jar -P ./jars
+wget https://repo1.maven.org/maven2/io/delta/delta-storage/2.2.0/delta-storage-2.2.0.jar -P ./jars
+cp gluten-XXXXX-spark-3.3-jar-with-dependencies.jar jars/
+```
+
+#### Query local data
+
+##### Start Spark Thriftserver on local
+```
+cd spark-3.2.2-bin-hadoop2.7
+./sbin/start-thriftserver.sh \
+ --master local[3] \
+ --driver-memory 10g \
+ --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.default.parallelism=1 \
+ --conf spark.sql.shuffle.partitions=1 \
+ --conf spark.sql.files.minPartitionNum=1 \
+ --conf spark.sql.files.maxPartitionBytes=1073741824 \
+ --conf spark.sql.adaptive.enabled=false \
+ --conf spark.locality.wait=0 \
+ --conf spark.locality.wait.node=0 \
+ --conf spark.locality.wait.process=0 \
+ --conf spark.sql.columnVector.offheap.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=6442450944 \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.executorEnv.LD_PRELOAD=/path_to_clickhouse_library/libch.so\
+ --conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.io.compression.codec=snappy \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.execution.datasources.v2.clickhouse.ClickHouseSparkCatalog \
+ --conf spark.databricks.delta.maxSnapshotLineageLength=20 \
+ --conf spark.databricks.delta.snapshotPartitions=1 \
+ --conf spark.databricks.delta.properties.defaults.checkpointInterval=5 \
+ --conf spark.databricks.delta.stalenessLimit=3600000
+
+#connect to Spark Thriftserver by beeline
+bin/beeline -u jdbc:hive2://localhost:10000/ -n root
+```
+
+##### Query local MergeTree files
+
+- Prepare data<span id="data-preparation"></span>
+
+Currently, the feature of writing ClickHouse MergeTree parts by Spark is developing, so you need to use command 'clickhouse-local' to generate MergeTree parts data manually. We provide a python script to call the command 'clickhouse-local' to convert parquet data to MergeTree parts:
+
+```
+
+#install ClickHouse community version
+sudo apt-get install -y apt-transport-https ca-certificates dirmngr
+sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
+echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
+sudo apt-get update
+sudo apt install -y --allow-downgrades clickhouse-server=22.5.1.2079 clickhouse-client=22.5.1.2079 clickhouse-common-static=22.5.1.2079
+
+#generate MergeTree parts
+mkdir -p /path_clickhouse_database/table_path/
+python3 /path_to_clickhouse_backend_src/utils/local-engine/tool/parquet_to_mergetree.py --path=/tmp --source=/path_to_parquet_data/tpch-data-sf100/lineitem --dst=/path_clickhouse_database/table_path/lineitem
+```
+
+**This python script will convert one parquet data file to one MergeTree parts.**
+
+- Create a TPC-H lineitem table using ClickHouse DataSource
+
+```
+ DROP TABLE IF EXISTS lineitem;
+ CREATE TABLE IF NOT EXISTS lineitem (
+ l_orderkey bigint,
+ l_partkey bigint,
+ l_suppkey bigint,
+ l_linenumber bigint,
+ l_quantity double,
+ l_extendedprice double,
+ l_discount double,
+ l_tax double,
+ l_returnflag string,
+ l_linestatus string,
+ l_shipdate date,
+ l_commitdate date,
+ l_receiptdate date,
+ l_shipinstruct string,
+ l_shipmode string,
+ l_comment string)
+ USING clickhouse
+ TBLPROPERTIES (engine='MergeTree'
+ )
+ LOCATION '/path_clickhouse_database/table_path/lineitem';
+```
+
+- TPC-H Q6 test
+
+```
+ SELECT
+ sum(l_extendedprice * l_discount) AS revenue
+ FROM
+ lineitem
+ WHERE
+ l_shipdate >= date'1994-01-01'
+ AND l_shipdate < date'1994-01-01' + interval 1 year
+ AND l_discount BETWEEN 0.06 - 0.01 AND 0.06 + 0.01
+ AND l_quantity < 24;
+```
+
+- Result
+
+ The DAG is shown on Spark UI as below:
+
+ 
+
+##### Query local Parquet files
+
+You can query local parquet files directly.
+```sql
+-- query on a single file
+select * from parquet.`/your_data_root_dir/1.parquet`;
+
+-- query on a directly which has multiple files
+select * from parquet.`/your_data_roo_dir/`;
+```
+
+You can also create a TEMPORARY VIEW for parquet files.
+```sql
+create or replace temporary view your_table_name
+using org.apache.spark.sql.parquet
+options(
+ path "/your_data_root_dir/"
+)
+```
+
+
+##### Query Parquet files in S3
+
+If you have parquet files in S3(either AWS S3 or S3 compatible storages like MINIO), you can query them directly.
+You need to add these additional configs to spark:
+
+```bash
+ --config spark.hadoop.fs.s3a.endpoint=S3_ENDPOINT
+ --config spark.hadoop.fs.s3a.path.style.access=true
+ --config spark.hadoop.fs.s3a.access.key=YOUR_ACCESS_KEY
+ --config spark.hadoop.fs.s3a.secret.key=YOUR_SECRET_KEY
+```
+where S3_ENDPOINT must follow the format of `https://s3.region-code.amazonaws.com`, e.g. `https://s3.us-east-1.amazonaws.com` (or `http://hostname:39090 for MINIO)
+
+When you query the parquet files in S3, you need to add the prefix `s3a://` to the path, e.g. `s3a://your_bucket_name/path_to_your_parquet`.
+
+Additionally, you can add these configs to enable local caching of S3 data. Each spark executor will have its own cache. Cache stealing between executors is not supported yet.
+
+```bash
+ --config spark.gluten.sql.columnar.backend.ch.runtime_config.s3.local_cache.enabled=true
+ --config spark.gluten.sql.columnar.backend.ch.runtime_config.s3.local_cache.cache_path=/executor_local_folder_for_cache
+```
+
+##### Use beeline to execute queries
+
+After start a spark thriftserver, we can use the beeline to connect to this server.
+```bash
+# run a file
+/path_to_spark/bin/beeline -u jdbc:hive2://localhost:10000 -f <your_sql_file>
+
+# run a query
+/path_to_spark/bin/beeline -u jdbc:hive2://localhost:10000 -e '<your_sql>'
+
+# enter a interactive mode
+/path_to_spark/bin/beeline -u jdbc:hive2://localhost:10000
+```
+
+#### Query Hive tables in HDFS
+
+Suppose that you have set up hive and hdfs, you can query the data on hive directly.
+
+- Copy `hive-site.xml` into `/path_to_spark/conf/`
+- Copy `hdfs-site.xml` into `/path_to_spark/conf/`, and edit `spark-env.sh`
+
+```bash
+# add this line into spark-env.sh
+export HADOOP_CONF_DIR=/path_to_spark/conf
+```
+
+- Start spark thriftserver with hdfs configurations
+
+```bash
+hdfs_conf_file=/your_local_path/hdfs-site.xml
+
+cd spark-3.2.2-bin-hadoop2.7
+# add a new option: spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf
+./sbin/start-thriftserver.sh \
+ --master local[3] \
+ --files $hdfs_conf_file \
+ --driver-memory 10g \
+ --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.default.parallelism=1 \
+ --conf spark.sql.shuffle.partitions=1 \
+ --conf spark.sql.files.minPartitionNum=1 \
+ --conf spark.sql.files.maxPartitionBytes=1073741824 \
+ --conf spark.locality.wait=0 \
+ --conf spark.locality.wait.node=0 \
+ --conf spark.locality.wait.process=0 \
+ --conf spark.sql.columnVector.offheap.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=6442450944 \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.executorEnv.LD_PRELOAD=/path_to_clickhouse_library/libch.so\
+ --conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.io.compression.codec=snappy \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.execution.datasources.v2.clickhouse.ClickHouseSparkCatalog \
+ --conf spark.databricks.delta.maxSnapshotLineageLength=20 \
+ --conf spark.databricks.delta.snapshotPartitions=1 \
+ --conf spark.databricks.delta.properties.defaults.checkpointInterval=5 \
+ --conf spark.databricks.delta.stalenessLimit=3600000 \
+ --conf spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf=./hdfs-site.xml
+```
+
+For example, you have a table `demo_database`.`demo_table` on the hive, you can run queries as below.
+
+```sql
+select * from demo_database.demo_talbe;
+```
+
+
+### Gluten in YARN cluster
+
+We can to run a Spark SQL task by gluten on a yarn cluster as following
+```bash
+#!/bin/bash
+
+# The file contains the sql you want to run
+sql_file=/path/to/spark/sql/file
+
+export SPARK_HOME=/path/to/spark/home
+spark_cmd=$SPARK_HOME/bin/spark-sql
+
+# Define the path to libch.so
+ch_lib=/path/to/libch.so
+export LD_PRELOAD=$ch_lib
+
+# copy gluten jar file to $SPARK_HOME/jar
+gluten_jar=/path/to/gluten/jar/file
+cp $gluten_jar $SPARK_HOME/jar
+
+batchsize=20480
+hdfs_conf=/path/to/hdfs-site.xml
+
+$spark_cmd \
+ --name gluten_on_yarn
+ --master yarn \
+ --deploy-mode client \
+ --files $ch_lib \
+ --executor-cores 1 \
+ --num-executors 2 \
+ --executor-memory 10g \
+ --conf spark.default.parallelism=4 \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=7g \
+ --conf spark.driver.maxResultSize=2g \
+ --conf spark.sql.autoBroadcastJoinThreshold=-1 \
+ --conf spark.sql.parquet.columnarReaderBatchSize=${batchsize} \
+ --conf spark.sql.inMemoryColumnarStorage.batchSize=${batchsize} \
+ --conf spark.sql.execution.arrow.maxRecordsPerBatch=${batchsize} \
+ --conf spark.sql.broadcastTimeout=4800 \
+ --conf spark.task.maxFailures=1 \
+ --conf spark.excludeOnFailure.enabled=false \
+ --conf spark.driver.maxResultSize=4g \
+ --conf spark.sql.adaptive.enabled=false \
+ --conf spark.dynamicAllocation.executorIdleTimeout=0s \
+ --conf spark.sql.shuffle.partitions=112 \
+ --conf spark.sql.sources.useV1SourceList=avro \
+ --conf spark.sql.files.maxPartitionBytes=1073741824 \
+ --conf spark.gluten.sql.columnar.loadnative=true \
+ --conf spark.gluten.sql.columnar.libpath=$ch_lib \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf=$hdfs_conf \
+ --conf spark.gluten.sql.columnar.backend.ch.runtime_config.logger.level=debug \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.executorEnv.LD_PRELOAD=$LD_PRELOAD \
+ --conf spark.hadoop.input.connect.timeout=600000 \
+ --conf spark.hadoop.input.read.timeout=600000 \
+ --conf spark.hadoop.input.write.timeout=600000 \
+ --conf spark.hadoop.dfs.client.log.severity="DEBUG2" \
+ --files $ch_lib \
+ -f $sql_file
+```
+
+We also can use `spark-submit` to run a task.
+
+
+### Benchmark with TPC-H 100 Q6 on Gluten with ClickHouse backend
+
+This benchmark is tested on AWS EC2 cluster, there are 7 EC2 instances:
+
+| Node Role | EC2 Type | Instances Count | Resources | AMI |
+| ---------- | ----------- | ------------- | ------------- | ------------- |
+| Master | m5.4xlarge | 1 | 16 cores 64G memory per node | ubuntu-focal-20.04 |
+| Worker | m5.4xlarge | 1 | 16 cores 64G memory per node | ubuntu-focal-20.04 |
+
+
+#### Deploy on Cloud
+
+- Tested on Spark Standalone cluster, its resources are shown below:
+
+ | | CPU cores | Memory | Instances Count |
+ | ---------- | ----------- | ------------- | ------------- |
+ | Spark Worker | 15 | 60G | 6 |
+
+- Prepare jars
+
+ Refer to [Deploy Spark 3.2.2](#deploy-spark-322)
+
+- Deploy gluten-substrait-XXXXX-jar-with-dependencies.jar
+
+```
+ #deploy 'gluten-substrait-XXXXX-jar-with-dependencies.jar' to every node, and then
+ cp gluten-substrait-XXXXX-jar-with-dependencies.jar /path_to_spark/jars/
+```
+
+- Deploy ClickHouse library
+
+ Deploy ClickHouse library 'libch.so' to every worker node.
+
+
+##### Deploy JuiceFS
+
+- JuiceFS uses Redis to save metadata, install redis firstly:
+
+```
+ wget https://download.redis.io/releases/redis-6.0.14.tar.gz
+ sudo apt install build-essential
+ tar -zxvf redis-6.0.14.tar.gz
+ cd redis-6.0.14
+ make
+ make install PREFIX=/home/ubuntu/redis6
+ cd ..
+ rm -rf redis-6.0.14
+
+ #start redis server
+ /home/ubuntu/redis6/bin/redis-server /home/ubuntu/redis6/redis.conf
+
+```
+- Use JuiceFS to format a S3 bucket and mount a volumn on every node
+
+ Please refer to [The-JuiceFS-Command-Reference](https://juicefs.com/docs/community/command_reference)
+
+```
+ wget https://github.com/juicedata/juicefs/releases/download/v0.17.5/juicefs-0.17.5-linux-amd64.tar.gz
+ tar -zxvf juicefs-0.17.5-linux-amd64.tar.gz
+
+ ./juicefs format --block-size 4096 --storage s3 --bucket https://s3.cn-northwest-1.amazonaws.com.cn/s3-gluten-tpch100/ --access-key "XXXXXXXX" --secret-key "XXXXXXXX" redis://:123456@master-ip:6379/1 gluten-tables
+
+ #mount a volumn on every node
+ ./juicefs mount -d --no-usage-report --no-syslog --attr-cache 7200 --entry-cache 7200 --dir-entry-cache 7200 --buffer-size 500 --prefetch 1 --open-cache 86400 --log /home/ubuntu/juicefs-logs/mount1.log --cache-dir /home/ubuntu/juicefs-cache/ --cache-size 102400 redis://:123456@master-ip:6379/1 /home/ubuntu/gluten/gluten_table
+ #create a directory for lineitem table path
+ mkdir -p /home/ubuntu/gluten/gluten_table/lineitem
+
+```
+
+#### Preparation
+
+Please refer to [Data-preparation](#data-preparation) to generate MergeTree parts data to the lineitem table path: /home/ubuntu/gluten/gluten_table/lineitem.
+
+#### Run Spark Thriftserver
+
+```
+cd spark-3.2.2-bin-hadoop2.7
+./sbin/start-thriftserver.sh \
+ --master spark://master-ip:7070 --deploy-mode client \
+ --driver-memory 16g --driver-cores 4 \
+ --total-executor-cores 90 --executor-memory 60g --executor-cores 15 \
+ --conf spark.driver.memoryOverhead=8G \
+ --conf spark.default.parallelism=90 \
+ --conf spark.sql.shuffle.partitions=1 \
+ --conf spark.sql.files.minPartitionNum=1 \
+ --conf spark.sql.files.maxPartitionBytes=536870912 \
+ --conf spark.sql.parquet.filterPushdown=true \
+ --conf spark.sql.parquet.enableVectorizedReader=true \
+ --conf spark.locality.wait=0 \
+ --conf spark.locality.wait.node=0 \
+ --conf spark.locality.wait.process=0 \
+ --conf spark.sql.columnVector.offheap.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=42949672960 \
+ --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.io.compression.codec=snappy \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.execution.datasources.v2.clickhouse.ClickHouseSparkCatalog \
+ --conf spark.databricks.delta.maxSnapshotLineageLength=20 \
+ --conf spark.databricks.delta.snapshotPartitions=1 \
+ --conf spark.databricks.delta.properties.defaults.checkpointInterval=5 \
+ --conf spark.databricks.delta.stalenessLimit=3600000
+```
+
+#### Test TPC-H Q6 with JMeter
+
+- Create a lineitem table using clickhouse datasource
+
+```
+ DROP TABLE IF EXISTS lineitem;
+ CREATE TABLE IF NOT EXISTS lineitem (
+ l_orderkey bigint,
+ l_partkey bigint,
+ l_suppkey bigint,
+ l_linenumber bigint,
+ l_quantity double,
+ l_extendedprice double,
+ l_discount double,
+ l_tax double,
+ l_returnflag string,
+ l_linestatus string,
+ l_shipdate date,
+ l_commitdate date,
+ l_receiptdate date,
+ l_shipinstruct string,
+ l_shipmode string,
+ l_comment string)
+ USING clickhouse
+ TBLPROPERTIES (engine='MergeTree'
+ )
+ LOCATION 'file:///home/ubuntu/gluten/gluten_table/lineitem';
+```
+
+- Run TPC-H Q6 test with JMeter
+ 1. Run TPC-H Q6 test 100 times in the first round;
+ 2. Run TPC-H Q6 test 1000 times in the second round;
+
+#### Performance
+
+The performance of Gluten + ClickHouse backend increases by **about 1/3**.
+
+| | 70% | 80% | 90% | 99% | Avg |
+| ---------- | ----------- | ------------- | ------------- | ------------- | ------------- |
+| Spark + Parquet | 590ms | 592ms | 597ms | 609ms | 588ms |
+| Spark + Gluten + ClickHouse backend | 402ms | 405ms | 409ms | 425ms | 399ms |
+
+
+### New CI System
+
+https://opencicd.kyligence.com/job/Gluten/job/gluten-ci/
+public read-only account：gluten/hN2xX3uQ4m
+
+### Celeborn support
+
+Gluten with clickhouse backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x`, `0.4.x` and `0.5.x`.
+
+Below introduction is used to enable this feature.
+
+First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster.
+
+When compiling the Gluten Java module, it's required to enable `celeborn` profile, as follows:
+
+```
+mvn clean package -Pbackends-clickhouse -Pspark-3.3 -Pceleborn -DskipTests
+```
+
+Then add the Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
+
+Currently to use Gluten following configurations are required in `spark-defaults.conf`
+
+```
+spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+spark.shuffle.service.enabled false
+
+# options: hash, sort
+# Hash shuffle writer use (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
+# Sort shuffle writer uses less memory than hash shuffle writer, if your shuffle partition count is large, try to use sort hash writer.
+spark.celeborn.client.spark.shuffle.writer hash
+
+# We recommend setting spark.celeborn.client.push.replicate.enabled to true to enable server-side data replication
+# If you have only one worker, this setting must be false
+# If your Celeborn is using HDFS, it's recommended to set this setting to false
+spark.celeborn.client.push.replicate.enabled true
+
+# Support for Spark AQE only tested under Spark 3
+# we recommend setting localShuffleReader to false to get better performance of Celeborn
+spark.sql.adaptive.localShuffleReader.enabled false
+
+# If Celeborn is using HDFS
+spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
+
+# If you want to use dynamic resource allocation,
+# please refer to this URL (https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+spark.dynamicAllocation.enabled false
+```
+
+
diff --git a/_docs/v1.5.0/get-started/GlutenUI.md b/_docs/v1.5.0/get-started/GlutenUI.md
new file mode 100644
index 0000000..8b556fa
--- /dev/null
+++ b/_docs/v1.5.0/get-started/GlutenUI.md
@@ -0,0 +1,37 @@
+---
+layout: page
+title: GlutenUI
+nav_order: 2
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+# Gluten UI
+
+## Gluten event
+
+Gluten provides two events `GlutenBuildInfoEvent` and `GlutenPlanFallbackEvent`:
+
+- GlutenBuildInfoEvent, it contains the Gluten build information so that we are able to be aware of the environment when doing some debug.
+ It includes `Java Version`, `Scala Version`, `GCC Version`, `Gluten Version`, `Spark Version`, `Hadoop Version`, `Gluten Revision`, `Backend`, `Backend Revision`, etc.
+
+- GlutenPlanFallbackEvent, it contains the fallback information for each query execution.
+ Note, if the query execution is in AQE, then Gluten will post it for each stage.
+
+Developers can register `SparkListener` to handle these two Gluten events.
+
+## SQL tab
+
+Gluten provides a tab based on Spark UI, named `Gluten SQL / DataFrame`
+
+
+
+This tab contains two parts:
+
+1. The Gluten build information.
+2. SQL/Dataframe queries fallback information.
+
+If you want to disable Gluten UI, add a config when submitting `--conf spark.gluten.ui.enabled=false`.
+
+## History server
+
+Gluten UI also supports Spark history server. Add gluten-ui jar into the history server classpath, e.g., $SPARK_HOME/jars, then restart history server.
diff --git a/_docs/v1.5.0/get-started/Velox.md b/_docs/v1.5.0/get-started/Velox.md
new file mode 100644
index 0000000..6388420
--- /dev/null
+++ b/_docs/v1.5.0/get-started/Velox.md
@@ -0,0 +1,597 @@
+---
+layout: page
+title: Velox
+nav_order: 3
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+# Supported Version
+
+| Type | Version |
+|-------|------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.5 |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
+
+# Prerequisite
+
+Currently, with static build Gluten+Velox backend supports all the Linux OSes, but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8** and their variants.
+
+Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4 and 3.5.5.
+
+We need to set up the `JAVA_HOME` env. Currently, Gluten supports **java 8** and **java 17**.
+
+**For x86_64**
+
+```bash
+## make sure jdk8 is used
+export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+export PATH=$JAVA_HOME/bin:$PATH
+```
+
+**For aarch64**
+
+```bash
+## make sure jdk8 is used
+export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64
+export PATH=$JAVA_HOME/bin:$PATH
+```
+
+**Get gluten**
+
+```bash
+## config maven, like proxy in ~/.m2/settings.xml
+
+## fetch gluten code
+git clone https://github.com/apache/incubator-gluten.git
+```
+
+# Build Gluten with Velox Backend
+
+It's recommended to use buildbundle-veloxbe.sh to build gluten in one script.
+[Gluten build guide](./build-guide.md) listed the parameters and their default value of build command for your reference.
+
+**For x86_64 build**
+
+First time build for all supported spark versions.
+
+```bash
+./dev/buildbundle-veloxbe.sh
+```
+
+After a complete build, if only some gluten code is changed, you can use the following command to skip building arrow and
+setting up build dependencies.
+
+```bash
+./dev/buildbundle-veloxbe.sh --build_arrow=OFF --run_setup_script=OFF
+```
+
+**For aarch64 build**
+
+```bash
+export CPU_TARGET="aarch64"
+
+./dev/buildbundle-veloxbe.sh
+```
+
+**Step-by-step build**
+
+Alternative to the above one-step build, you can follow the below guide for step-by-step build.
+Currently, Gluten is using a [forked Velox](https://github.com/oap-project/velox/) which is daily updated based on [upstream Velox](https://github.com/facebookincubator/velox).
+
+```bash
+
+# Build arrow with some patches applied. We need these slight code changes till arrow is upgraded
+# to 17.0.0 or newer versions.
+./dev/builddeps-veloxbe.sh build_arrow
+
+./dev/builddeps-veloxbe.sh build_velox
+
+./dev/builddeps-veloxbe.sh build_gluten_cpp
+
+## compile Gluten java module and create package jar
+cd /path/to/gluten
+# For spark3.2.x
+mvn clean package -Pbackends-velox -Pspark-3.2 -DskipTests
+# For spark3.3.x
+mvn clean package -Pbackends-velox -Pspark-3.3 -DskipTests
+# For spark3.4.x
+mvn clean package -Pbackends-velox -Pspark-3.4 -DskipTests
+# For spark3.5.x
+mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests
+```
+
+Notes： Building Velox may fail caused by OOM. You can prevent this failure by adjusting `NUM_THREADS` (e.g., `export NUM_THREADS=4`) before building Gluten/Velox. The recommended minimal memory size is 64G.
+
+After the above build process, the Jar file will be generated under `package/target/`.
+
+Alternatively you may refer to [build in docker](../developers/velox-backend-build-in-docker.md) to build the gluten jar in docker.
+
+## Dependency library deployment
+
+With build option `enable_vcpkg=ON`, all dependency libraries will be statically linked to `libvelox.so` and `libgluten.so` which are packed into the gluten-jar.
+In this way, only the gluten-jar is needed to add to `spark.<driver|executor>.extraClassPath` and spark will deploy the jar to each worker node. It's better to build
+the static version using a clean docker image without any extra libraries installed ( [build in docker](../developers/velox-backend-build-in-docker.md) ). On host with
+some libraries like jemalloc installed, the script may crash with odd message. You may need to uninstall those libraries to get a clean host. We **strongly recommend** user to build Gluten in this way to avoid dependency lacking issue.
+
+With build option `enable_vcpkg=OFF`, not all dependency libraries will be dynamically linked. After building, you need to separately execute `./dev/build-thirdparty.sh` to
+pack required shared libraries into another jar named `gluten-thirdparty-lib-$LINUX_OS-$VERSION-$ARCH.jar`. Then you need to add the jar to Spark config `extraClassPath` and
+set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install required shared libraries with **exactly the same versions** on each worker node . You may find the
+libraries list from the third-party jar.
+
+# Remote storage support
+
+## HDFS support
+
+Gluten supports dynamically loading both libhdfs.so and libhdfs3.so at runtime by using dlopen, allowing the JVM to load the appropriate shared library file as needed. This means you do not need to set the library path during the compilation phase.
+To enable this functionality, you must set the JAVA_HOME and HADOOP_HOME environment variables. Gluten will then locate and load the ${HADOOP_HOME}/lib/native/libhdfs.so file at runtime. If you prefer to use libhdfs3.so instead, simply replace the ${HADOOP_HOME}/lib/native/libhdfs.so file with libhdfs3.so.
+
+### Build libhdfs3
+
+If you want to run Gluten with libhdfs3.so, you need to manually compile libhdfs3 to obtain the libhdfs3.so file. We provide the script dev/build_libhdfs3.sh in Gluten to help you compile libhdfs3.so.
+
+### Build with HDFS support
+
+To build Gluten with HDFS support, below command is suggested:
+
+```bash
+cd /path/to/gluten
+./dev/buildbundle-veloxbe.sh --enable_hdfs=ON
+```
+
+### Configuration about HDFS support in Libhdfs3
+
+HDFS uris (hdfs://host:port) will be extracted from a valid hdfs file path to initialize hdfs client, you do not need to specify it explicitly.
+
+libhdfs3 need a configuration file and [example here](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/src/backend/utils/misc/etc/hdfs-client.xml), this file is a bit different from hdfs-site.xml and core-site.xml.
+Download that example config file to local and do some needed modifications to support HA or else, then set env variable like below to use it, or upload it to HDFS to use, more details [here](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/depends/libhdfs3/src/client/Hdfs.cpp#L171-L189).
+
+```
+// Spark local mode
+export LIBHDFS3_CONF="/path/to/hdfs-client.xml"
+
+// Spark Yarn cluster mode
+--conf spark.executorEnv.LIBHDFS3_CONF="/path/to/hdfs-client.xml"
+
+// Spark Yarn cluster mode and upload hdfs config file
+cp /path/to/hdfs-client.xml hdfs-client.xml
+--files hdfs-client.xml
+```
+
+One typical deployment on Spark/HDFS cluster is to enable [short-circuit reading](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html). Short-circuit reads provide a substantial performance boost to many applications.
+
+By default libhdfs3 does not set the default hdfs domain socket path to support HDFS short-circuit read. If this feature is required in HDFS setup, users may need to setup the domain socket path correctly by patching the libhdfs3 source code or by setting the correct config environment. In Gluten the short-circuit domain socket path is set to "/var/lib/hadoop-hdfs/dn_socket" in [build_velox.sh](https://github.com/apache/incubator-gluten/blob/main/ep/build-velox/src/build_velox.sh) So we [...]
+
+```
+sudo mkdir -p /var/lib/hadoop-hdfs/
+sudo chown <sparkuser>:<sparkuser> /var/lib/hadoop-hdfs/
+```
+
+You also need to add configuration to the "hdfs-site.xml" as below:
+
+```
+<property>
+ <name>dfs.client.read.shortcircuit</name>
+ <value>true</value>
+</property>
+<property>
+ <name>dfs.domain.socket.path</name>
+ <value>/var/lib/hadoop-hdfs/dn_socket</value>
+</property>
+```
+
+### Kerberos support in libhdfs3
+
+Here are two steps to enable kerberos.
+
+- Make sure the hdfs-client.xml contains
+
+```
+<property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+</property>
+```
+
+- Specify the environment variable [KRB5CCNAME](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/depends/libhdfs3/src/client/FileSystem.cpp#L56) and upload the kerberos ticket cache file
+
+```
+--conf spark.executorEnv.KRB5CCNAME=krb5cc_0000 --files /tmp/krb5cc_0000
+```
+
+The ticket cache file can be found by `klist`.
+
+## Azure Blob File System (ABFS) support
+
+Velox supports ABFS with the open source [Azure SDK for C++](https://github.com/Azure/azure-sdk-for-cpp) and Gluten uses the Velox ABFS connector to connect with ABFS.
+The build option for ABFS (enable_abfs) must be set to enable this feature as listed below.
+
+```
+cd /path/to/gluten
+./dev/buildbundle-veloxbe.sh --enable_abfs=ON
+```
+
+Please refer [Velox ABFS](VeloxABFS.md) part for more detailed configurations.
+
+
+## AWS S3 support
+
+Velox supports S3 with the open source [AWS C++ SDK](https://github.com/aws/aws-sdk-cpp) and Gluten uses Velox S3 connector to connect with S3.
+A new build option for S3(enable_s3) is added. Below command is used to enable this feature
+
+```
+cd /path/to/gluten
+./dev/buildbundle-veloxbe.sh --enable_s3=ON
+```
+
+Currently there are several ways to access S3 in Spark. Please refer [Velox S3](VeloxS3.md) part for more detailed configurations
+
+## GCS support
+
+Please refer [GCS](VeloxGCS.md)
+
+# Remote Shuffle Service Support
+
+## Celeborn support
+
+Gluten with velox backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x`, `0.4.x` and `0.5.x`.
+
+Below introduction is used to enable this feature.
+
+First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster.
+
+When compiling the Gluten Java module, it's required to enable `celeborn` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Pceleborn -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath (usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
+- Gluten: gluten-velox-bundle-spark3.x_2.12-xxx.jar (The bundled Gluten JAR built with the -Pceleborn profile specified)
+
+Currently, to use Gluten, the following configurations are required in `spark-defaults.conf`
+
+```
+spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+spark.shuffle.service.enabled false
+
+# options: hash, sort
+# Hash shuffle writer use (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
+# Sort shuffle writer uses less memory than hash shuffle writer, if your shuffle partition count is large, try to use sort hash writer.
+spark.celeborn.client.spark.shuffle.writer hash
+
+# We recommend setting spark.celeborn.client.push.replicate.enabled to true to enable server-side data replication
+# If you have only one worker, this setting must be false
+# If your Celeborn is using HDFS, it's recommended to set this setting to false
+spark.celeborn.client.push.replicate.enabled true
+
+# Support for Spark AQE only tested under Spark 3
+# we recommend setting localShuffleReader to false to get better performance of Celeborn
+spark.sql.adaptive.localShuffleReader.enabled false
+
+# If Celeborn is using HDFS
+spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
+
+# If you want to use dynamic resource allocation,
+# please refer to this URL (https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+spark.dynamicAllocation.enabled false
+```
+
+Additionally, for sort-based shuffle, Celeborn supports two types of shuffle writers: the default row-based sort shuffle writer and the RSS sort shuffle writer.
+By default, Celeborn uses the RSS sort shuffle writer. You can switch to the default row-based sort shuffle writer
+by setting the following configuration:
+
+```
+spark.gluten.sql.columnar.shuffle.celeborn.useRssSort false
+```
+
+## Uniffle support
+
+Uniffle with velox backend supports [Uniffle](https://github.com/apache/incubator-uniffle) as remote shuffle service. Currently, the supported Uniffle versions are `0.9.2`.
+
+First refer to this URL(https://uniffle.apache.org/docs/intro) to get start with uniffle.
+
+When compiling the Gluten Java module, it's required to enable `uniffle` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Puniffle -DskipTests
+```
+
+Then add the Uniffle and Spark Celeborn Client packages to your Spark application's classpath (usually add them into `$SPARK_HOME/jars`).
+
+- Uniffle: rss-client-spark3-shaded-[uniffleVersion].jar
+- Gluten: gluten-velox-bundle-spark3.x_2.12-xxx.jar (The bundled Gluten JAR built with the -Puniffle profile specified)
+
+Currently, to use Gluten, the following configurations are required in `spark-defaults.conf`
+
+```
+spark.shuffle.manager org.apache.spark.shuffle.gluten.uniffle.UniffleShuffleManager
+
+# uniffle coordinator address
+spark.rss.coordinator.quorum ip:port
+
+# Support for Spark AQE
+spark.sql.adaptive.localShuffleReader.enabled false
+spark.shuffle.service.enabled false
+
+# Uniffle support mutilple storage types, you can choose one of them.
+# Such as MEMORY,LOCALFILE,MEMORY_LOCALFILE,HDFS,MEMORY_HDFS,LOCALFILE_HDFS,MEMORY_LOCALFILE_HDFS
+spark.rss.storage.type LOCALFILE_HDFS
+
+# If you want to use dynamic resource allocation,
+# please refer to this URL (https://uniffle.apache.org/docs/client-guide#support-spark-dynamic-allocation) for more details.
+spark.dynamicAllocation.enabled false
+```
+
+# Datalake Framework Support
+
+## DeltaLake Support
+
+Gluten with velox backend supports [DeltaLake](https://delta.io/) table.
+
+### How to use
+
+First of all, compile gluten-delta module by a `delta` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Pdelta -DskipTests
+```
+
+Once built successfully, delta features will be included in gluten-velox-bundle-X jar. Then you can query delta table by gluten/velox without scan's fallback.
+
+Gluten with velox backends also support the column mapping of delta tables.
+About column mapping, see more [here](https://docs.delta.io/latest/delta-column-mapping.html).
+
+## Iceberg Support
+
+Gluten with velox backend supports [Iceberg](https://iceberg.apache.org/) table. Currently, both reading COW (Copy-On-Write) and MOR (Merge-On-Read) tables are supported.
+
+### How to use
+
+First of all, compile gluten-iceberg module by a `iceberg` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests
+```
+
+Once built successfully, iceberg features will be included in gluten-velox-bundle-X jar. Then you can query iceberg table by gluten/velox without scan's fallback.
+
+## Hudi Support
+
+Gluten with velox backend supports [Hudi](https://hudi.apache.org/) table. Currently, only reading COW (Copy-On-Write) tables is supported.
+
+## Paimon Support
+
+Gluten with velox backend supports [Paimon](https://paimon.apache.org/) table. Currently, only non-pk table is supported, and the Spark version needs to be >= 3.3.
+
+### How to use
+
+Compile gluten-paimon module by a `paimon` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.5 -Ppaimon -DskipTests
+```
+
+Once built successfully, paimon features will be included in gluten-velox-bundle-X jar. Then you can query paimon non-pk table by gluten/velox without scan's fallback.
+
+### How to use
+
+First of all, compile gluten-hudi module by a `hudi` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Phudi -DskipTests
+```
+
+Once built successfully, hudi features will be included in gluten-velox-bundle-X jar. Then you can query hudi **COW** table by gluten/velox without scan's fallback.
+
+# Coverage
+
+Spark3.3 has 387 functions in total. ~240 are commonly used. To get the support status of all Spark built-in functions, please refer to [Velox Backend's Supported Operators & Functions](../velox-backend-support-progress.md).
+
+> Velox doesn't support [ANSI mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)), so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will fallback to Vanilla Spark.
+
+To identify what can be offloaded in a query and detailed fallback reasons, user can follow below steps to retrieve corresponding logs.
+
+```
+1) Enable Gluten by proper [configuration](https://github.com/apache/incubator-gluten/blob/main/docs/Configuration.md).
+
+2) Disable Spark AQE to trigger plan validation in Gluten
+spark.sql.adaptive.enabled = false
+
+3) Check physical plan
+sparkSession.sql("your_sql").explain()
+```
+
+With above steps, you will get a physical plan output like:
+
+```
+== Physical Plan ==
+-Execute InsertIntoHiveTable (7)
+ +- Coalesce (6)
+ +- VeloxColumnarToRowExec (5)
+ +- ^ ProjectExecTransformer (3)
+ +- GlutenRowToArrowColumnar (2)
+ +- Scan hive default.table (1)
+
+```
+
+`GlutenRowToArrowColumnar`/`VeloxColumnarToRowExec` indicates there is a fallback operator before or after it. And you may find fallback reason like below in logs.
+
+```
+native validation failed due to: in ProjectRel, Scalar function name not registered: get_struct_field, called with arguments: (ROW<col_0:INTEGER,col_1:BIGINT,col_2:BIGINT>, INTEGER).
+```
+
+In the above, the symbol `^` indicates a plan is offloaded to Velox in a stage. In Spark DAG, all such pipelined plans (consecutive plans marked with `^`) are plotted
+inside an umbrella node named `WholeStageCodegenTransformer` (It's not codegen node. The naming is just for making it well plotted like Spark Whole Stage Codegen).
+
+# Spill
+
+Velox backend supports spilling-to-disk.
+
+Using the following configuration options to customize spilling:
+
+| Name | Default Value | Description |
+|--------------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| spark.gluten.sql.columnar.backend.velox.spillStrategy | auto | none: Disable spill on Velox backend; auto: Let Spark memory manager manage Velox's spilling |
+| spark.gluten.sql.columnar.backend.velox.spillFileSystem | local | The filesystem used to store spill data. local: The local file system. heap-over-local: Write files to JVM heap if having extra heap space. Otherwise write to local file system. |
+| spark.gluten.sql.columnar.backend.velox.aggregationSpillEnabled | true | Whether spill is enabled on aggregations |
+| spark.gluten.sql.columnar.backend.velox.joinSpillEnabled | true | Whether spill is enabled on joins |
+| spark.gluten.sql.columnar.backend.velox.orderBySpillEnabled | true | Whether spill is enabled on sorts |
+| spark.gluten.sql.columnar.backend.velox.maxSpillLevel | 4 | The max allowed spilling level with zero being the initial spilling level |
+| spark.gluten.sql.columnar.backend.velox.maxSpillFileSize | 1GB | The max allowed spill file size. If it is zero, then there is no limit |
+| spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit | 48 | The start partition bit which is used with 'spillPartitionBits' together to calculate the spilling partition number |
+| spark.gluten.sql.columnar.backend.velox.spillPartitionBits | 2 | The number of bits used to calculate the spilling partition number. The number of spilling partitions will be power of two |
+| spark.gluten.sql.columnar.backend.velox.spillableReservationGrowthPct | 25 | The spillable memory reservation growth percentage of the previous memory reservation size |
+| spark.gluten.sql.columnar.backend.velox.spillThreadNum | 0 | (Experimental) The thread num of a dedicated thread pool to do spill
+
+# Velox User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAF)
+
+Please check the [VeloxNativeUDF.md](../developers/VeloxUDF.md) for more detailed usage and configurations.
+
+# Test TPC-H or TPC-DS on Gluten with Velox backend
+
+All TPC-H and TPC-DS queries are supported in Gluten Velox backend. You may refer to the [notebook](../../tools/workload/benchmark_velox) we used to do the performance test.
+
+## Data preparation
+
+The data generation scripts are [TPC-H dategen script](../../tools/workload/tpch/gen_data/parquet_dataset/tpch_datagen_parquet.sh) and
+[TPC-DS dategen script](../../tools/workload/tpcds/gen_data/parquet_dataset/tpcds_datagen_parquet.sh).
+
+The used TPC-H and TPC-DS queries are the original ones, and can be accessed from [TPC-DS queries](../../tools/gluten-it/common/src/main/resources/tpcds-queries)
+and [TPC-H queries](../../tools/gluten-it/common/src/main/resources/tpch-queries).
+
+## Submit the Spark SQL job
+
+Submit test script from spark-shell. You can find the scala code to [Run TPC-H](../../tools/workload/tpch/run_tpch/tpch_parquet.scala) as an example. Please remember to modify
+the location of TPC-H files as well as TPC-H queries before you run the testing.
+
+```
+var parquet_file_path = "/PATH/TO/TPCH_PARQUET_PATH"
+var gluten_root = "/PATH/TO/GLUTEN"
+```
+
+Below script shows an example about how to run the testing, you should modify the parameters such as executor cores, memory, offHeap size based on your environment.
+
+```bash
+export GLUTEN_JAR = /PATH/TO/GLUTEN/package/target/<gluten-jar>
+cat tpch_parquet.scala | spark-shell --name tpch_powertest_velox \
+ --master yarn --deploy-mode client \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=20g \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --num-executors 6 \
+ --executor-cores 6 \
+ --driver-memory 20g \
+ --executor-memory 25g \
+ --conf spark.executor.memoryOverhead=5g \
+ --conf spark.driver.maxResultSize=32g
+```
+
+Refer to [Gluten configuration](../Configuration.md) for more details.
+
+## Result
+
+*wholestagetransformer* indicates that the offloading works.
+
+
+
+## Performance
+
+Below table shows the TPC-H Q1 and Q6 Performance in a multiple-thread test (--num-executors 6 --executor-cores 6) for Velox and vanilla Spark.
+Both Parquet and ORC datasets are sf1024.
+
+| Query Performance (s) | Velox (ORC) | Vanilla Spark (Parquet) | Vanilla Spark (ORC) |
+|---------------- | ----------- | ------------- | ------------- |
+| TPC-H Q6 | 13.6 | 21.6 | 34.9 |
+| TPC-H Q1 | 26.1 | 76.7 | 84.9 |
+
+# Gluten UI
+
+Please refer [Gluten UI](GlutenUI.md)
+
+# Gluten Native Plan Summary
+
+## Gluten Implicits
+Gluten provides a helper class to get the fallback summary from a Spark Dataset.
+
+```
+import org.apache.spark.sql.execution.GlutenImplicits._
+val df = spark.sql("SELECT * FROM t")
+df.fallbackSummary
+```
+
+Note that, if AQE is enabled, but the query is not materialized, then it will re-plan
+the query execution with disabled AQE. It is a workaround to get the final plan, and it may
+cause the inconsistent results with a materialized query. However, we have no choice.
+
+## Native Plan in Spark's Explain Output
+
+Gluten supports inject native plan string into Spark explain with formatted mode by setting `--conf spark.gluten.sql.injectNativePlanStringToExplain=true`.
+Here is an example, how Gluten shows the native plan string.
+
+```
+(9) WholeStageCodegenTransformer (2)
+Input [6]: [c1#0L, c2#1L, c3#2L, c1#3L, c2#4L, c3#5L]
+Arguments: false
+Native Plan:
+-- Project[expressions: (n3_6:BIGINT, "n0_0"), (n3_7:BIGINT, "n0_1"), (n3_8:BIGINT, "n0_2"), (n3_9:BIGINT, "n1_0"), (n3_10:BIGINT, "n1_1"), (n3_11:BIGINT, "n1_2")] -> n3_6:BIGINT, n3_7:BIGINT, n3_8:BIGINT, n3_9:BIGINT, n3_10:BIGINT, n3_11:BIGINT
+ -- HashJoin[INNER n1_1=n0_1] -> n1_0:BIGINT, n1_1:BIGINT, n1_2:BIGINT, n0_0:BIGINT, n0_1:BIGINT, n0_2:BIGINT
+ -- TableScan[table: hive_table, range filters: [(c2, Filter(IsNotNull, deterministic, null not allowed))]] -> n1_0:BIGINT, n1_1:BIGINT, n1_2:BIGINT
+ -- ValueStream[] -> n0_0:BIGINT, n0_1:BIGINT, n0_2:BIGINT
+```
+
+## Native Plan with Stats
+
+Gluten supports print native plan with statistics to executor system output stream by setting
+`--conf spark.gluten.sql.columnar.backend.velox.showTaskMetricsWhenFinished=true` or `--conf spark.gluten.sql.debug=true`.
+Note that the plan string with statistics is task level, which may increase the size of the executor logs.
+Below is an example of how Gluten displays the native plan string with statistics.
+
+```
+I20231121 10:19:42.348845 90094332 WholeStageResultIterator.cc:220] Native Plan with stats for: [Stage: 1 TID: 16]
+-- Project[expressions: (n3_6:BIGINT, "n0_0"), (n3_7:BIGINT, "n0_1"), (n3_8:BIGINT, "n0_2"), (n3_9:BIGINT, "n1_0"), (n3_10:BIGINT, "n1_1"), (n3_11:BIGINT, "n1_2")] -> n3_6:BIGINT, n3_7:BIGINT, n3_8:BIGINT, n3_9:BIGINT, n3_10:BIGINT, n3_11:BIGINT
+ Output: 27 rows (3.56KB, 3 batches), Cpu time: 10.58us, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+ queuedWallNanos sum: 2.00us, count: 1, min: 2.00us, max: 2.00us
+```
+
+
+## Using Stage-Level Resource Adjustment to Avoid OOM(Experimental)
+ see more [here](./VeloxStageResourceAdj.md)
+
+## Broadcast Build Relations to Off-Heap(Experimental)
+
+The experimental feature **Off-Heap Broadcast Build Relations** aims to mitigate out-of-memory (OOM) issues caused by heap memory consumption during broadcast operations. Detailed design
+can be found [here](https://docs.google.com/document/d/1eZNWPUEdiz2JPJfhyVn9hrk6SqJFRNzOMZm6u5Yredk/edit?tab=t.0)
+
+### Purpose & how it works
+- **Avoid OOM**: Prevent OOM errors when broadcasting large datasets.
+- **Reduce Heap Memory Usage**: Store broadcast build relations in Spark off-heap memory instead of on-heap memory
+
+### Configuration
+
+### Enable Off-Heap Broadcast
+To enable this feature, you can set the following Spark configuration:
+
+| Property | Default | Description |
+|-------------------------------------------------------------|---------|-------------------------------------------------------------------|
+| `spark.gluten.velox.offHeapBroadcastBuildRelation.enabled` | `false` | Enable/disable off-heap storage for broadcast build relations. |
+
+This feature has been tested through a series of tests, and we are collecting more feedback from users. If you have memory problem on broadcast build relations, please try this feature and give more feedbacks.
+
+**Note**: This feature will become the default behavior once stabilized. Stay tuned for updates!
+
+
+# Accelerators
+
+Please refer [QAT](VeloxQAT.md) for details
diff --git a/_docs/v1.5.0/get-started/VeloxABFS.md b/_docs/v1.5.0/get-started/VeloxABFS.md
new file mode 100644
index 0000000..233600b
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxABFS.md
@@ -0,0 +1,24 @@
+---
+layout: page
+title: VeloxABFS
+nav_order: 4
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+ABFS is an important data store for big data users. This doc discusses config details and use cases of Gluten with ABFS. To use an ABFS account as your data source, please ensure you use the listed ABFS config in your spark-defaults.conf. If you would like to authenticate with ABFS using additional auth mechanisms, please reach out using the 'Issues' tab.
+
+# Working with ABFS
+
+## Configuring ABFS Access Token
+
+To configure access to your storage account, replace <storage-account> with the name of your account. This property aligns with Spark configurations. By setting this config multiple times using different storage account names, you can access multiple ABFS accounts.
+
+```sh
+spark.hadoop.fs.azure.account.key.<storage-account>.dfs.core.windows.net XXXXXXXXX
+```
+
+### Other authentatication methods are not yet supported.
+
+# Local Caching support
+
+Velox supports a local cache when reading data from ABFS. Please refer [Velox Local Cache](VeloxLocalCache.md) part for more detailed configurations.
\ No newline at end of file
diff --git a/_docs/v1.5.0/get-started/VeloxGCS.md b/_docs/v1.5.0/get-started/VeloxGCS.md
new file mode 100644
index 0000000..8d86373
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxGCS.md
@@ -0,0 +1,54 @@
+---
+layout: page
+title: VeloxGCS
+nav_order: 5
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+Object stores offered by CSPs such as GCS are important for users of Gluten to store their data. This doc will discuss all details of configs, and use cases around using Gluten with object stores. In order to use a GCS endpoint as your data source, please ensure you are using the following GCS configs in your spark-defaults.conf. If you're experiencing any issues authenticating to GCS with additional auth mechanisms, please reach out to us using the 'Issues' tab.
+
+# Working with GCS
+
+## Installing the gcloud CLI
+
+To access GCS Objects using Gluten and Velox, first you have to [download an install the gcloud CLI] (https://cloud.google.com/sdk/docs/install).
+
+
+## Configuring GCS using a user account
+
+This is recommended for regular users, follow the [instructions to authorize a user account](https://cloud.google.com/sdk/docs/authorizing#user-account).
+After these steps, no specific configuration is required for Gluten, since the authorization was handled entirely by the gcloud tool.
+
+
+## Configuring GCS using a credential file
+
+For workloads that need to be fully automated, manually authorizing can be problematic. For such cases it is better to use a json file with the credentials.
+This is described in the [instructions to configure a service account]https://cloud.google.com/sdk/docs/authorizing#service-account.
+
+Such json file with the credentials can be passed to Gluten:
+
+```sh
+spark.hadoop.fs.gs.auth.type SERVICE_ACCOUNT_JSON_KEYFILE
+spark.hadoop.fs.gs.auth.service.account.json.keyfile // path to the json file with the credentials.
+```
+
+## Configuring GCS endpoints
+
+For cases when a GCS mock is used, an optional endpoint can be provided:
+```sh
+spark.hadoop.fs.gs.storage.root.url // url to the mock gcs service including starting with http or https
+```
+
+## Configuring GCS max retry count
+
+For cases when a transient server error is detected, GCS can be configured to keep retrying until a number of transient error is detected.
+```sh
+spark.hadoop.fs.gs.http.max.retry // number of times to keep retrying unless a non-transient error is detected
+```
+
+## Configuring GCS max retry time
+
+For cases when a transient server error is detected, GCS can be configured to keep retrying until the retry loop exceeds a prescribed duration.
+```sh
+spark.hadoop.fs.gs.http.max.retry-time // a string representing the time keep retrying (10s, 1m, etc).
+```
\ No newline at end of file
diff --git a/_docs/v1.5.0/get-started/VeloxGPU.md b/_docs/v1.5.0/get-started/VeloxGPU.md
new file mode 100644
index 0000000..2351688
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxGPU.md
@@ -0,0 +1,81 @@
+---
+layout: page
+title: VeloxGPU
+nav_order: 6
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+# GPU Acceleration in Velox/Gluten
+*Unified execution engine leveraging CUDF for hardware-accelerated Spark SQL queries*
+
+---
+
+## **1. Overview**
+- **Purpose**: Accelerate Velox operators via CUDF APIs, replacing CPU execution when enabled.
+- **Status**: Experimental (TPC-H SF1 validated). Integrates RAPIDS ecosystem with Apache Spark via Gluten .
+- **Key Benefit**: Some queries achieved up to **8.1x speedup** on x86 vs. Spark Java engine .
+
+---
+
+## **2. Prerequisites**
+- **CUDA Toolkit**: 12.8.0 ([download](https://developer.nvidia.com/cuda-downloads?target_os=Linux)).
+- **NVIDIA Drivers**: Compatible with CUDA 12.8.
+- **Container Toolkit**: Install `nvidia-container-toolkit` ([guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)).
+- **System Reboot**: Required after driver installation.
+- **Environment Setup**: Use [`start_cudf.sh`](https://github.com/apache/incubator-gluten/tree/main/dev/start_cudf.sh) for host configuration .
+
+---
+
+## **3. Implementation Mechanics**
+- **Operator Conversion**:
+ - Velox PlanNodes → **GPU operators** when `spark.gluten.sql.columnar.cudf=true`.
+ - Falls back to CPU operators if GPU unsupported (triggers row/columnar data conversion) .
+- **Debugging**: Enable `spark.gluten.debug.enabled.cudf=true` for operator replacement logs.
+- **Memory**: Global [RMM](https://docs.rapids.ai/api/librmm/stable/) memory manager, cannot align with Spark memory system.
+
+---
+
+## **4. Docker Deployment**
+```bash
+docker pull apache/gluten:centos-9-jdk8-cudf # Pre-built GPU image
+docker run --name gpu_gluten_container --gpus all -it apache/gluten:centos-9-jdk8-cudf
+```
+- **Image Includes**: Native build cache, Gluten dependencies, Spark 3.4 environment.
+
+---
+
+## **5. Build & Deployment**
+#### **Dependencies**
+The OS, Spark version, Java version aligns with Gluten CPU.
+
+### **Compilation Commands**
+If building in the docker image, no need to set up script and build arrow.
+```bash
+./dev/buildbundle-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF --enable_cudf=ON
+```
+
+---
+
+## **6. GPU Operator Support Status**
+| **Operator** | **Status** | **Notes** |
+|-----------------|-----------------|--------------------------|
+| **Scan** | ❌ Not supported| In Development |
+| **Project** | ⚠️ Partial | Function TPCH-compatible |
+| **Filter** | ✅ Implemented | Core operator |
+| **OrderBy** | ✅ Implemented | Merged in Velox #12735 |
+| **Aggregation** | ⚠️ Partial | TPCH-compatible |
+| **Join** | ⚠️ Partial | TPCH-compatible |
+| **Spill** | ❌ Not supported | In Planning |
+
+---
+
+## **7. Performance Validation**
+
+GPU performs better on operator HashJoin and HashAggregation.
+Single Operator like Hash Agg shows 5x speedup.
+
+---
+
+## **8. Relevant Resources**
+1. [CUDF Docs](https://docs.rapids.ai/api/cudf/stable/libcudf_docs/) - GPU operator APIs.
+2. [Gluten GPU Issue #9098](https://github.com/apache/incubator-gluten/issues/8851) - Development tracker.
diff --git a/_docs/v1.5.0/get-started/VeloxIceberg.md b/_docs/v1.5.0/get-started/VeloxIceberg.md
new file mode 100644
index 0000000..9d78c51
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxIceberg.md
@@ -0,0 +1,120 @@
+---
+layout: page
+title: VeloxIceberg
+nav_order: 7
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+# Iceberg Support in Velox Backend
+
+## Supported Spark version
+
+All the spark version is supported, but for convenience, only Spark 3.4 is well tested.
+Now only read is supported in Gluten.
+
+## Support Status
+Following value indicates the iceberg support progress:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Offload to the Velox backend |
+| PartialOffload | Some operators offload and some fallback |
+| Fallback | Fallback to spark to execute |
+| Exception | Cannot fallback by some conditions, throw the exception |
+| ResultMismatch | Some hidden bug may cause result mismatch, especially for some corner case |
+
+## Adding catalogs
+Fallback
+
+## Creating a table
+Fallback
+
+## Writing
+Fallback
+````
+INSERT INTO local.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+PartialOffload
+
+The write is fallback while read is offload.
+````
+INSERT INTO local.db.table SELECT id, data FROM source WHERE length(data) = 1;
+````
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table Type | No Delete | Position Delete | Equality Delete |
+|-------------|-----------------|-----------------|-----------------|
+| unpartition | Offload | Offload | Fallback |
+| partition | Fallback mostly | Fallback mostly | Fallback |
+| metadata | Fallback | | |
+
+Offload the simple query.
+````
+SELECT count(1) as count, data
+FROM local.db.table
+GROUP BY data;
+````
+
+If delete by Spark and copy on read, will generate position delete file, the query may offload.
+
+If delete by Flink, may generate the equality delete file, fallback in tht case.
+
+Now we only offload the simple query, for partition table, many operators are fallback by Expression
+StaticInvoke such as BucketFunction, wait to be supported.
+
+DataFrame reads are supported and can now reference tables by name using spark.table:
+
+````
+val df = spark.table("local.db.table")
+df.count()
+````
+
+### Read metadata
+Fallback
+````
+SELECT data, _file FROM local.db.table;
+````
+
+## DataType
+Timestamptz in orc format is not supported, throws exception.
+UUID type and Fixed type is fallback.
+
+## Format
+PartialOffload
+
+Supports parquet and orc format.
+Not support avro format.
+
+## SQL
+Only support SELECT.
+
+## Schema evolution
+PartialOffload
+
+Gluten uses column name to match the parquet file, so if the column is renamed or
+the added column name is same to the deleted column, the scan will fall back.
+
+## Configuration
+### Catalogs
+Supports all the catalog options, which is not used in native engine.
+
+### SQL Extensions
+Fallback
+
+Supports the option `spark.sql.extensions`, fallback the SQL command `CALL`.
+
+### Runtime configuration
+#### Read options
+
+| Spark option | Status |
+|-----------------------|-------------|
+| snapshot-id | Support |
+| as-of-timestamp | Support |
+| split-size | Support |
+| lookback | Support |
+| file-open-cost | Support |
+| vectorization-enabled | Not Support |
+| batch-size | Not Support |
diff --git a/_docs/v1.5.0/get-started/VeloxLocalCache.md b/_docs/v1.5.0/get-started/VeloxLocalCache.md
new file mode 100644
index 0000000..e12fb97
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxLocalCache.md
@@ -0,0 +1,20 @@
+---
+layout: page
+title: VeloxLocalCache
+nav_order: 8
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+Velox supports a local cache when reading data from HDFS/S3/ABFS. With this feature, Velox can asynchronously cache the data on local disk when reading from remote storage and future read requests on previously cached blocks will be serviced from local cache files. To enable the local caching feature, the following configurations are required:
+
+```
+spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable velox cache, default false.
+spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of in-mem cache, default is 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to store the cache files, default is "/tmp".
+spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of the SSD cache, default is 128MB. Velox will do in-mem cache only if this value is 0.
+spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the SSD cache, default is 1.
+spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the IO threads for cache promoting, default is 1. Velox will try to do "read-ahead" if this value is bigger than 1
+spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable O_DIRECT on cache write, default false.
+```
+
+It's recommended to mount SSDs to the cache path to get the best performance of local caching. Cache files will be written to "spark.gluten.sql.columnar.backend.velox.cachePath", with UUID based suffix, e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten cannot reuse older caches for now, and the old cache files are left after Spark context shutdown.
diff --git a/_docs/v1.5.0/get-started/VeloxQAT.md b/_docs/v1.5.0/get-started/VeloxQAT.md
new file mode 100644
index 0000000..cefdbda
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxQAT.md
@@ -0,0 +1,155 @@
+---
+layout: page
+title: VeloxQAT
+nav_order: 9
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+# Intel® QuickAssist Technology (QAT) support
+
+Gluten supports using Intel® QuickAssist Technology (QAT) for data compression during Spark Shuffle. It benefits from QAT Hardware-based acceleration on compression/decompression, and uses Gzip as compression format for higher compression ratio to reduce the pressure on disks and network transmission.
+
+This feature is based on QAT driver library and [QATzip](https://github.com/intel/QATzip) library. Please manually download QAT driver for your system, and follow its README to build and install on all Driver and Worker node: [Intel® QuickAssist Technology Driver for Linux* – HW Version 2.0](https://www.intel.com/content/www/us/en/download/765501/intel-quickassist-technology-driver-for-linux-hw-version-2-0.html?wapkw=quickassist).
+
+## Software Requirements
+
+- Download QAT driver for your system, and follow its README to build and install on all Driver and Worker nodes: [Intel® QuickAssist Technology Driver for Linux* – HW Version 2.0](https://www.intel.com/content/www/us/en/download/765501/intel-quickassist-technology-driver-for-linux-hw-version-2-0.html?wapkw=quickassist).
+- Below compression libraries need to be installed on all Driver and Worker nodes:
+ - Zlib* library of version 1.2.7 or higher
+ - ZSTD* library of version 1.5.4 or higher
+ - LZ4* library
+
+## Build Gluten with QAT
+
+1. Setup ICP_ROOT environment variable to the directory where QAT driver is extracted. This environment variable is required during building Gluten and running Spark applications. It's recommended to put it in .bashrc on Driver and Worker nodes.
+
+```bash
+echo "export ICP_ROOT=/path/to/QAT_driver" >> ~/.bashrc
+source ~/.bashrc
+
+# Also set for root if running as non-root user
+sudo su -
+echo "export ICP_ROOT=/path/to/QAT_driver" >> ~/.bashrc
+exit
+```
+
+2. **This step is required if your application is running as Non-root user.**
+ The users must be added to the 'qat' group after QAT drvier is installed. And change the amount of max locked memory for the username that is included in the group name. This can be done by specifying the limit in /etc/security/limits.conf.
+
+```bash
+sudo su -
+usermod -aG qat username # need relogin to take effect
+
+# To set 500MB add a line like this in /etc/security/limits.conf
+echo "@qat - memlock 500000" >> /etc/security/limits.conf
+
+exit
+```
+
+3. Enable huge page. This step is required to execute each time after system reboot. We recommend using systemctl to manage at system startup. You change the values for "max_huge_pages" and "max_huge_pages_per_process" to make sure there are enough resources for your workload. As for Spark applications, one process matches one executor. Within the executor, every task is allocated a maximum of 5 huge pages.
+
+```bash
+sudo su -
+
+cat << EOF > /usr/local/bin/qat_startup.sh
+#!/bin/bash
+echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+rmmod usdm_drv
+insmod $ICP_ROOT/build/usdm_drv.ko max_huge_pages=1024 max_huge_pages_per_process=32
+EOF
+
+chmod +x /usr/local/bin/qat_startup.sh
+
+cat << EOF > /etc/systemd/system/qat_startup.service
+[Unit]
+Description=Configure QAT
+
+[Service]
+ExecStart=/usr/local/bin/qat_startup.sh
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl enable qat_startup.service
+systemctl start qat_startup.service # setup immediately
+systemctl status qat_startup.service
+
+exit
+```
+
+4. After the setup, you are now ready to build Gluten with QAT. Use the command below to enable this feature:
+
+```bash
+cd /path/to/gluten
+
+## The script builds four jars for spark 3.2.2, 3.3.1, 3.4.3 and 3.5.1.
+./dev/buildbundle-veloxbe.sh --enable_qat=ON
+```
+
+## Enable QAT with Gzip/Zstd for shuffle compression
+
+1. To offload shuffle compression into QAT, first make sure you have the right QAT configuration file at /etc/4xxx_devX.conf. We provide a [example configuration file](../qat/4x16.conf). This configuration sets up to 4 processes that can bind to 1 QAT, and each process can use up to 16 QAT DC instances.
+
+```bash
+## run as root
+## Overwrite QAT configuration file.
+cd /etc
+for i in {0..7}; do echo "4xxx_dev$i.conf"; done | xargs -i cp -f /path/to/gluten/docs/qat/4x16.conf {}
+## Restart QAT after updating configuration files.
+adf_ctl restart
+```
+
+2. Check QAT status and make sure the status is up
+
+```bash
+adf_ctl status
+```
+
+The output should be like:
+
+```
+Checking status of all devices.
+There is 8 QAT acceleration device(s) in the system:
+ qat_dev0 - type: 4xxx, inst_id: 0, node_id: 0, bsf: 0000:6b:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev1 - type: 4xxx, inst_id: 1, node_id: 1, bsf: 0000:70:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev2 - type: 4xxx, inst_id: 2, node_id: 2, bsf: 0000:75:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev3 - type: 4xxx, inst_id: 3, node_id: 3, bsf: 0000:7a:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev4 - type: 4xxx, inst_id: 4, node_id: 4, bsf: 0000:e8:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev5 - type: 4xxx, inst_id: 5, node_id: 5, bsf: 0000:ed:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev6 - type: 4xxx, inst_id: 6, node_id: 6, bsf: 0000:f2:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev7 - type: 4xxx, inst_id: 7, node_id: 7, bsf: 0000:f7:00.0, #accel: 1 #engines: 9 state: up
+```
+
+3. Extra Gluten configurations are required when starting Spark application
+
+```
+--conf spark.gluten.sql.columnar.shuffle.codec=gzip # Valid options are gzip and zstd
+--conf spark.gluten.sql.columnar.shuffle.codecBackend=qat
+```
+
+4. You can use below command to check whether QAT is working normally at run-time. The value of fw_counters should continue to increase during shuffle.
+
+```
+while :; do cat /sys/kernel/debug/qat_4xxx_0000:6b:00.0/fw_counters; sleep 1; done
+```
+
+## QAT driver references
+
+**Documentation**
+
+[README Text Files (README_QAT20.L.1.0.0-00021.txt)](https://downloadmirror.intel.com/765523/README_QAT20.L.1.0.0-00021.txt)
+
+**Release Notes**
+
+Check out the [Intel® QuickAssist Technology Software for Linux*](https://www.intel.com/content/www/us/en/content-details/632507/intel-quickassist-technology-intel-qat-software-for-linux-release-notes-hardware-version-2-0.html) - Release Notes for the latest changes in this release.
+
+**Getting Started Guide**
+
+Check out the [Intel® QuickAssist Technology Software for Linux*](https://www.intel.com/content/www/us/en/content-details/632506/intel-quickassist-technology-intel-qat-software-for-linux-getting-started-guide-hardware-version-2-0.html) - Getting Started Guide for detailed installation instructions.
+
+**Programmer's Guide**
+
+Check out the [Intel® QuickAssist Technology Software for Linux*](https://www.intel.com/content/www/us/en/content-details/743912/intel-quickassist-technology-intel-qat-software-for-linux-programmers-guide-hardware-version-2-0.html) - Programmer's Guide for software usage guidelines.
+
+For more Intel® QuickAssist Technology resources go to [Intel® QuickAssist Technology (Intel® QAT)](https://developer.intel.com/quickassist)
diff --git a/_docs/v1.5.0/get-started/VeloxS3.md b/_docs/v1.5.0/get-started/VeloxS3.md
new file mode 100644
index 0000000..99f4cfe
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxS3.md
@@ -0,0 +1,69 @@
+---
+layout: page
+title: VeloxS3
+nav_order: 10
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+Object stores offered by CSPs such as AWS S3 are important for users of Gluten to store their data. This doc will discuss all details of configs, and use cases around using Gluten with object stores. In order to use an S3 endpoint as your data source, please ensure you are using the following S3 configs in your spark-defaults.conf. If you're experiencing any issues authenticating to S3 with additional auth mechanisms, please reach out to us using the 'Issues' tab.
+
+# Working with S3
+
+## Configuring S3 endpoint
+
+S3 provides the endpoint based method to access the files, here's the example configuration. Users may need to modify some values based on real setup.
+
+```sh
+spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
+spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
+spark.hadoop.fs.s3a.access.key XXXXXXXXX
+spark.hadoop.fs.s3a.secret.key XXXXXXXXX
+spark.hadoop.fs.s3a.endpoint https://s3.us-west-1.amazonaws.com
+spark.hadoop.fs.s3a.connection.ssl.enabled true
+spark.hadoop.fs.s3a.path.style.access false
+```
+
+## Configuring S3 instance credentials
+
+S3 also provides other methods for accessing, you can also use instance credentials by setting the following config
+
+```
+spark.hadoop.fs.s3a.use.instance.credentials true
+```
+Note that in this case, "spark.hadoop.fs.s3a.endpoint" won't take affect as Gluten will use the endpoint set during instance creation.
+
+## Configuring S3 IAM roles
+You can also use iam role credentials by setting the following configurations. Instance credentials have higher priority than iam credentials.
+
+```
+spark.hadoop.fs.s3a.iam.role xxxx
+spark.hadoop.fs.s3a.iam.role.session.name xxxx
+```
+
+Note that `spark.hadoop.fs.s3a.iam.role.session.name` is optional.
+
+## Other authentatication methods are not supported yet
+
+- [AWS temporary credential](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html)
+
+## Log granularity of AWS C++ SDK in velox
+
+You can change log granularity of AWS C++ SDK by setting the `spark.gluten.velox.awsSdkLogLevel` configuration. The Allowed values are:
+ "OFF", "FATAL", "ERROR", "WARN", "INFO", "DEBUG", "TRACE".
+
+## Configuring Whether To Use Proxy From Env for S3 C++ Client
+You can change whether to use proxy from env for S3 C++ client by setting the `spark.gluten.velox.s3UseProxyFromEnv` configuration. The Allowed values are:
+ "false", "true".
+
+## Configuring S3 Payload Signing Policy
+You can change the S3 payload signing policy by setting the `spark.gluten.velox.s3PayloadSigningPolicy` configuration. The Allowed values are:
+ "Always", "RequestDependent", "Never".
+- When set to "Always", the payload checksum is included in the signature calculation.
+- When set to "RequestDependent", the payload checksum is included based on the value returned by "AmazonWebServiceRequest::SignBody()".
+
+## Configuring S3 Log Location
+You can set the log location by setting the `spark.gluten.velox.s3LogLocation` configuration.
+
+# Local Caching support
+
+Velox supports a local cache when reading data from S3. Please refer [Velox Local Cache](VeloxLocalCache.md) part for more detailed configurations.
\ No newline at end of file
diff --git a/_docs/v1.5.0/get-started/VeloxStageResourceAdj.md b/_docs/v1.5.0/get-started/VeloxStageResourceAdj.md
new file mode 100644
index 0000000..3e2115c
--- /dev/null
+++ b/_docs/v1.5.0/get-started/VeloxStageResourceAdj.md
@@ -0,0 +1,76 @@
+---
+layout: page
+title: VeloxStageResourceAdj
+nav_order: 11
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+## Using Stage-Level Resource Adjustment to Avoid OOM(Experimental)
+
+### **Overview**
+One major advantage of Apache Gluten is its ability to significantly reduce memory requirements per executor—potentially by up to half—when entire stages are offloaded to the native engine. This engine primarily relies on off-heap memory with minimal on-heap usage. However, when stages contain fallback operators that utilize the JVM engine, the on-heap memory size must be increased, leading to even higher memory demands per executor. This challenge has posed significant barriers during t [...]
+
+To address this issue, Apache Gluten introduces a stage-level resource auto-adjustment framework. This feature dynamically optimizes task and executor resource profiles, such as heap and off-heap memory allocation, based on the specific characteristics of each stage, including the presence of fallback operators. Additionally, this framework is designed with future enhancements in mind, allowing for adjustments to accommodate other requirements, such as heavy shuffle workloads(to be suppo [...]
+
+### **Prerequisites**
+1. **Enable Adaptive Query Execution (AQE)**:
+ ```properties
+ spark.sql.adaptive.enabled=true
+ ```
+2. **Enable Executor Dynamic Allocation**:
+ ```properties
+ spark.dynamicAllocation.enabled=true
+ ```
+3. **Resource Scheduler Compatibility**:
+ Ensure the underlying cluster resource manager (e.g., YARN, Kubernetes) supports dynamic resource allocation.
+
+### **Key Configurations**
+Add the following configurations to your Spark application:
+
+
+| Parameters | Description | Default |
+|-------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| spark.gluten.auto.adjustStageResource.enabled | Experimental: If enabled, gluten will try to set the stage resource according to stage execution plan. NOTE: Only works when aqe is enabled at the same time. | false |
+| spark.gluten.auto.adjustStageResources.heap.ratio | Experimental: Increase executor heap memory when match adjust stage resource rule. | 2.0d |
+| spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold | Experimental: Increase executor heap memory when stage contains fallen node count exceeds the total node count ratio. | 0.5d |
+
+#### **1. Enable Auto-Adjustment**
+```properties
+spark.gluten.auto.AdjustStageResource.enabled=true
+```
+### **How It Works**
+The framework analyzes each stage during query planning and adjusts resource profiles in following scenarios:
+
+#### **Scenario 1: Fallback Operators Exist**
+If a stage all operator fallback to vanilla Spark operator or fallback operators (e.g., unsupported UDAFs) ratio exceed specified threshold, Gluten will automic increases heap memory allocation to handle the extra load.
+
+
+### **Verification**
+1. **Check Logs**:
+ Look for driver log entries indicating resource profile adjustments:
+ ```
+ Apply resource profile [RP_ID] for plan [plan node name]
+ ```
+
+2. **Check SparkUI SQL Tab**
+
+There will be a ApplyResourceProfile node in the Query details.
+
+And the execution plan will like following with ApplyResourceProfile node inserted.
+```
++- *(3) HashAggregate(keys=[_nondeterministic#37], functions=[count(1)], output=[java_method(java.lang.Integer, signum, c1)#35, count(1)#36L])
+ +- AQEShuffleRead coalesced
+ +- ShuffleQueryStage 0
+ +- Exchange hashpartitioning(_nondeterministic#37, 5), ENSURE_REQUIREMENTS, [plan_id=607]
+ +- ApplyResourceProfile Profile: id = 0, executor resources: cores -> name: cores, amount: 1, script: , vendor: ,memory -> name: memory, amount: 1024, script: , vendor: ,offHeap -> name: offHeap, amount: 2048, script: , vendor: , task resources: cpus -> name: cpus, amount: 1.0
+ +- *(2) HashAggregate(keys=[_nondeterministic#37], functions=[partial_count(1)], output=[_nondeterministic#37, count#41L])
+ +- Project [java_method(java.lang.Integer, signum, c1#22) AS _nondeterministic#37]
+ +- *(1) ColumnarToRow
+ +- FileScan parquet default.tmp1[c1#22] Batched: true, DataFilters: [], Format: Parquet
+```
+
+### **Limitations**
+• Tested with YARN/Kubernetes; other resource managers may need validation.
+
+
+For issues or feedback, refer to [GLUTEN-8018](https://github.com/apache/incubator-gluten/issues/8018).
\ No newline at end of file
diff --git a/_docs/v1.5.0/get-started/Work-with-pyspark.ipynb b/_docs/v1.5.0/get-started/Work-with-pyspark.ipynb
new file mode 100644
index 0000000..23d7c77
--- /dev/null
+++ b/_docs/v1.5.0/get-started/Work-with-pyspark.ipynb
@@ -0,0 +1,136 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "509d5a1e-10f2-4294-a104-9fdb85a29b0c",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "SLF4J: Class path contains multiple SLF4J bindings.\n",
+ "SLF4J: Found binding in [jar:file:/root/spark-3.3.1-bin-hadoop2/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\n",
+ "SLF4J: Found binding in [jar:file:/root/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\n",
+ "SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\n",
+ "SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]\n",
+ "24/04/03 22:04:56 WARN Utils: Your hostname, vsr542 resolves to a loopback address: 127.0.1.1; using 10.0.2.142 instead (on interface eno1)\n",
+ "24/04/03 22:04:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
+ "Setting default log level to \"WARN\".\n",
+ "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
+ "24/04/03 22:04:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
+ "24/04/03 22:05:00 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import findspark\n",
+ "findspark.init()\n",
+ "\n",
+ "from pyspark import SparkConf, SparkContext\n",
+ "nativesql_jars = \"/path/to/gluten-XXXX.jar\"\n",
+ "conf = SparkConf().setAppName(\"PySpark Gluten\").setMaster(\"yarn\")\n",
+ "conf.set(\"spark.executor.instances\", \"2\")\n",
+ "conf.set(\"spark.executor.memory\", \"6g\")\n",
+ "conf.set(\"spark.executor.cores\", \"2\")\n",
+ "conf.set(\"spark.driver.memory\", \"2g\")\n",
+ "conf.set(\"spark.memory.offHeap.enabled\", \"true\")\n",
+ "conf.set(\"spark.memory.offHeap.size\", \"2g\")\n",
+ "conf.set(\"spark.executor.memoryOverhead\", \"384M\")\n",
+ "conf.set(\"spark.driver.extraClassPath\", nativesql_jars)\n",
+ "conf.set(\"spark.executor.extraClassPath\", nativesql_jars)\n",
+ "conf.set(\"spark.plugins\", \"org.apache.gluten.GlutenPlugin\")\n",
+ "conf.set(\"spark.gluten.loadLibFromJar\", \"false\")\n",
+ "conf.set(\"spark.shuffle.manager\", \"org.apache.spark.shuffle.sort.ColumnarShuffleManager\")\n",
+ "sc = SparkContext(conf=conf)\n",
+ "from pyspark.sql import SparkSession\n",
+ "spark_session = SparkSession(sc)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "6c4ec66a-3b8c-4a65-b948-8420b492333d",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "== Physical Plan ==\n",
+ "VeloxColumnarToRowExec\n",
+ "+- ^(1) FilterExecTransformer (isnotnull(category#1) AND (category#1 = Cellphone))\n",
+ " +- ^(1) InputIteratorTransformer[product#0, category#1, revenue#2L]\n",
+ " +- ^(1) InputAdapter\n",
+ " +- ^(1) RowToVeloxColumnar\n",
+ " +- *(1) Scan ExistingRDD[product#0,category#1,revenue#2L]\n",
+ "\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "df = spark_session.createDataFrame(\n",
+ " [\n",
+ " (\"Normal\", \"Cellphone\", 6000),\n",
+ " (\"Normal\", \"Tablet\", 1500),\n",
+ " (\"Mini\", \"Tablet\", 5500),\n",
+ " (\"Mini\", \"Cellphone\", 5000),\n",
+ " (\"Foldable\", \"Cellphone\", 6500),\n",
+ " (\"Foldable\", \"Tablet\", 2500),\n",
+ " (\"Pro\", \"Cellphone\", 3000),\n",
+ " (\"Pro\", \"Tablet\", 4000),\n",
+ " (\"Pro Max\", \"Cellphone\", 4500)\n",
+ " ],\n",
+ " [\"product\", \"category\", \"revenue\"]\n",
+ ")\n",
+ "df.filter(\"category = 'Cellphone'\").explain()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.19"
+ },
+ "nbTranslate": {
+ "displayLangs": [
+ "*"
+ ],
+ "hotkey": "alt-t",
+ "langInMainMenu": true,
+ "sourceLang": "en",
+ "targetLang": "fr",
+ "useGoogleTranslate": true
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/_docs/v1.5.0/get-started/build-guide.md b/_docs/v1.5.0/get-started/build-guide.md
new file mode 100644
index 0000000..24d7dc0
--- /dev/null
+++ b/_docs/v1.5.0/get-started/build-guide.md
@@ -0,0 +1,76 @@
+---
+layout: page
+title: build guide
+nav_order: 12
+parent: Getting-Started
+grand_parent: v1.5.0
+---
+## Build Parameters
+### Native build parameters for buildbundle-veloxbe.sh or builddeps-veloxbe.sh
+Please set them via `--`, e.g. `--build_type=Release`.
+
+| Parameters | Description | Default |
+|------------------------|----------------------------------------------------------------------------------------------------|---------|
+| build_type | Build type for Velox & gluten cpp, CMAKE_BUILD_TYPE. | Release |
+| build_tests | Build gluten cpp tests. | OFF |
+| build_examples | Build udf example. | OFF |
+| build_benchmarks | Build gluten cpp benchmarks. | OFF |
+| enable_jemalloc_stats | Print jemalloc stats for debugging. | OFF |
+| enable_qat | Enable QAT for shuffle data de/compression. | OFF |
+| enable_s3 | Build with S3 support. | OFF |
+| enable_gcs | Build with GCS support. | OFF |
+| enable_hdfs | Build with HDFS support. | OFF |
+| enable_abfs | Build with ABFS support. | OFF |
+| enable_vcpkg | Enable vcpkg for static build. | OFF |
+| run_setup_script | Run setup script to install Velox dependencies. | ON |
+| velox_repo | Specify your own Velox repo to build. | "" |
+| velox_branch | Specify your own Velox branch to build. | "" |
+| velox_home | Specify your own Velox source path to build. | "" |
+| build_velox_tests | Build Velox tests. | OFF |
+| build_velox_benchmarks | Build Velox benchmarks (velox_tests and connectors will be disabled if ON) | OFF |
+| build_arrow | Build arrow java/cpp and install the libs in local. Can turn it OFF after first build. | ON |
+| spark_version | Build for specified version of Spark(3.2, 3.3, 3.4, 3.5, ALL). `ALL` means build for all versions. | ALL |
+
+### Velox build parameters for build_velox.sh
+Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
+
+| Parameters | Description | Default |
+|------------------|---------------------------------------------------------------|------------------------------------------|
+| velox_home | Specify Velox source path to build. | GLUTEN_SRC/ep/build-velox/build/velox_ep |
+| build_type | Velox build type, i.e., CMAKE_BUILD_TYPE. | Release |
+| enable_s3 | Build Velox with S3 support. | OFF |
+| enable_gcs | Build Velox with GCS support. | OFF |
+| enable_hdfs | Build Velox with HDFS support. | OFF |
+| enable_abfs | Build Velox with ABFS support. | OFF |
+| run_setup_script | Run setup script to install Velox dependencies before build. | ON |
+| build_test_utils | Build Velox with cmake arg -DVELOX_BUILD_TEST_UTILS=ON if ON. | OFF |
+| build_tests | Build Velox test. | OFF |
+| build_benchmarks | Build Velox benchmarks. | OFF |
+
+### Maven build parameters
+The below parameters can be set via `-P` for mvn.
+
+| Parameters | Description | Default state |
+|---------------------|---------------------------------------|---------------|
+| backends-velox | Build Gluten Velox backend. | disabled |
+| backends-clickhouse | Build Gluten ClickHouse backend. | disabled |
+| celeborn | Build Gluten with Celeborn. | disabled |
+| uniffle | Build Gluten with Uniffle. | disabled |
+| delta | Build Gluten with Delta Lake support. | disabled |
+| iceberg | Build Gluten with Iceberg support. | disabled |
+| hudi | Build Gluten with Hudi support. | disabled |
+| spark-3.2 | Build Gluten for Spark 3.2. | enabled |
+| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
+| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
+| spark-3.5 | Build Gluten for Spark 3.5. | disabled |
+
+## Gluten Jar for Deployment
+The gluten jar built out is under `GLUTEN_SRC/package/target/`.
+It's name pattern is `gluten-<backend_type>-bundle-spark<spark.bundle.version>_<scala.binary.version>-<os.detected.release>_<os.detected.release.version>-<project.version>.jar`.
+
+| Spark Version | spark.bundle.version | scala.binary.version |
+|---------------|----------------------|----------------------|
+| 3.2.2 | 3.2 | 2.12 |
+| 3.3.1 | 3.3 | 2.12 |
+| 3.4.4 | 3.4 | 2.12 |
+| 3.5.5 | 3.5 | 2.12 |
diff --git a/_docs/v1.5.0/get-started/index.md b/_docs/v1.5.0/get-started/index.md
new file mode 100644
index 0000000..61a46a0
--- /dev/null
+++ b/_docs/v1.5.0/get-started/index.md
@@ -0,0 +1,9 @@
+---
+layout: page
+title: Getting-Started
+nav_order: 1
+has_children: true
+parent: v1.5.0
+permalink: /archives/v1.5.0/getting-started
+---
+# Getting Started with Gluten for Apache Spark
diff --git a/_docs/v1.5.0/index.md b/_docs/v1.5.0/index.md
new file mode 100644
index 0000000..354c845
--- /dev/null
+++ b/_docs/v1.5.0/index.md
@@ -0,0 +1,9 @@
+---
+layout: page
+title: v1.5.0
+nav_order: 1
+has_children: true
+parent: Archives
+permalink: /archives/v1.5.0/
+---
+# Gluten Documents by version
diff --git a/_docs/v1.5.0/velox-backend/index.md b/_docs/v1.5.0/velox-backend/index.md
new file mode 100644
index 0000000..cbb66a0
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/index.md
@@ -0,0 +1,9 @@
+---
+layout: page
+title: Velox Backend
+nav_order: 2
+has_children: true
+parent: v1.5.0
+permalink: /archives/v1.5.0/velox-backend
+---
+# Gluten Documents by version
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-aggregate-function-support.md b/_docs/v1.5.0/velox-backend/velox-backend-aggregate-function-support.md
new file mode 100644
index 0000000..5b3bf5f
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-aggregate-function-support.md
@@ -0,0 +1,78 @@
+---
+layout: page
+title: velox backend aggregate function support
+nav_order: 1
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+# Aggregate Functions Support Status
+
+**Out of 62 aggregate functions in Spark 3.5, Gluten currently fully supports 54 functions and partially supports 1 function.**
+
+## Aggregate Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-----------------------|------------------------------------|----------|----------------|
+| any | BoolOr | S | |
+| any_value | AnyValue | S | |
+| approx_count_distinct | HyperLogLogPlusPlus | S | |
+| approx_percentile | ApproximatePercentile | S | |
+| array_agg | CollectList | S | |
+| avg | Average | S | |
+| bit_and | BitAndAgg | S | |
+| bit_or | BitOrAgg | S | |
+| bit_xor | BitXorAgg | S | |
+| bitmap_construct_agg | BitmapConstructAgg | | |
+| bitmap_or_agg | BitmapOrAgg | | |
+| bool_and | BoolAnd | S | |
+| bool_or | BoolOr | S | |
+| collect_list | CollectList | S | |
+| collect_set | CollectSet | S | |
+| corr | Corr | S | |
+| count | Count | S | |
+| count_if | CountIf | S | |
+| count_min_sketch | CountMinSketchAggExpressionBuilder | | |
+| covar_pop | CovPopulation | S | |
+| covar_samp | CovSample | S | |
+| every | BoolAnd | S | |
+| first | First | S | |
+| first_value | First | S | |
+| grouping | Grouping | S | |
+| grouping_id | GroupingID | S | |
+| histogram_numeric | HistogramNumeric | | |
+| hll_sketch_agg | HllSketchAgg | | |
+| hll_union_agg | HllUnionAgg | | |
+| kurtosis | Kurtosis | S | |
+| last | Last | S | |
+| last_value | Last | S | |
+| max | Max | S | |
+| max_by | MaxBy | S | |
+| mean | Average | S | |
+| median | Median | S | |
+| min | Min | S | |
+| min_by | MinBy | S | |
+| mode | Mode | | |
+| percentile | Percentile | S | |
+| percentile_approx | ApproximatePercentile | S | |
+| regr_avgx | RegrAvgX | S | |
+| regr_avgy | RegrAvgY | S | |
+| regr_count | RegrCount | S | |
+| regr_intercept | RegrIntercept | S | |
+| regr_r2 | RegrR2 | S | |
+| regr_slope | RegrSlope | S | |
+| regr_sxx | RegrSXX | S | |
+| regr_sxy | RegrSXY | S | |
+| regr_syy | RegrSYY | S | |
+| skewness | Skewness | S | |
+| some | BoolOr | S | |
+| std | StddevSamp | S | |
+| stddev | StddevSamp | S | |
+| stddev_pop | StddevPop | S | |
+| stddev_samp | StddevSamp | S | |
+| sum | Sum | S | |
+| try_avg | TryAverageExpressionBuilder | S | |
+| try_sum | TrySumExpressionBuilder | PS | |
+| var_pop | VariancePop | S | |
+| var_samp | VarianceSamp | S | |
+| variance | VarianceSamp | S | |
+
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-generator-function-support.md b/_docs/v1.5.0/velox-backend/velox-backend-generator-function-support.md
new file mode 100644
index 0000000..6b44554
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-generator-function-support.md
@@ -0,0 +1,23 @@
+---
+layout: page
+title: velox backend generator function support
+nav_order: 2
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+# Generator Functions Support Status
+
+**Out of 7 generator functions in Spark 3.5, Gluten currently fully supports 7 functions.**
+
+## Generator Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|--------------------------|----------|----------------|
+| explode | ExplodeExpressionBuilder | S | |
+| explode_outer | ExplodeExpressionBuilder | S | |
+| inline | Inline | S | |
+| inline_outer | Inline | S | |
+| posexplode | PosExplode | S | |
+| posexplode_outer | PosExplode | S | |
+| stack | Stack | S | |
+
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-limitations.md b/_docs/v1.5.0/velox-backend/velox-backend-limitations.md
new file mode 100644
index 0000000..0ded652
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-limitations.md
@@ -0,0 +1,174 @@
+---
+layout: page
+title: velox backend limitations
+nav_order: 3
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+This document describes the limitations of velox backend by listing some known cases where exception will be thrown, gluten behaves incompatibly with spark, or certain plan's execution
+must fall back to vanilla spark, etc.
+
+### Override of Spark classes (For Spark3.2 and Spark3.3)
+Gluten avoids to modify Spark's existing code and use Spark APIs if possible. However, some APIs aren't exposed in Vanilla spark and we have to copy the Spark file and do the hardcode changes. The list of override classes can be found as ignoreClasses in package/pom.xml . If you use customized Spark, you may check if the files are modified in your spark, otherwise your changes will be overrided.
+
+So you need to ensure preferentially load the Gluten jar to overwrite the jar of vanilla spark. Refer to [How to prioritize loading Gluten jars in Spark](https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-troubleshooting.md#incompatible-class-error-when-using-native-writer).
+
+If not officially supported spark3.2/3.3 version is used, NoSuchMethodError can be thrown at runtime. More details see [issue-4514](https://github.com/apache/incubator-gluten/issues/4514).
+
+### Fallbacks
+Except the unsupported operators, functions, file formats, data sources listed in , there are some known cases also fall back to Vanilla Spark.
+
+#### ANSI
+Gluten currently doesn't support ANSI mode. If ANSI is enabled, Spark plan's execution will always fall back to vanilla Spark.
+
+#### Runtime BloomFilter
+Velox BloomFilter's serialization format is different from Spark's. BloomFilter binary generated by Velox can't be deserialized by vanilla spark. So if `might_contain` falls back, we fall back `bloom_filter_agg` to vanilla spark also.
+
+#### Case Sensitive mode
+Gluten only supports spark default case-insensitive mode. If case-sensitive mode is enabled, user may get incorrect result.
+
+#### Regexp functions
+In Velox, regexp functions (`rlike`, `regexp_extract`, etc.) are implemented based on RE2, while in Spark they are based on `java.util.regex`.
+* Lookaround (lookahead/lookbehind) pattern is not supported in RE2.
+* When matching white space with pattern "\\s", RE2 doesn't treat "\v" (or "\x0b") as white space, but `java.util.regex` does.
+
+There are a few unknown incompatible cases. If user cannot tolerate the incompatibility risk, please enable the below configuration property.
+```
+spark.gluten.sql.fallbackRegexpExpressions
+```
+
+#### FileSource format
+Currently, Gluten only fully supports parquet file format and partially support ORC. If other format is used, scan operator falls back to vanilla spark.
+
+#### Partitioned Table Scan
+Gluten only support the partitioned table scan when the file path contain the partition info, otherwise will fall back to vanilla spark.
+
+### Incompatible behavior
+In certain cases, Gluten result may be different from Vanilla spark.
+
+#### JSON functions
+Velox only supports double quotes surrounded strings, not single quotes, in JSON data. If single quotes are used, gluten will produce incorrect result.
+
+Velox doesn't support [*] in path when get_json_object function is called and returns null instead.
+
+#### Parquet read conf
+Gluten supports `spark.files.ignoreCorruptFiles` with default false, if true, the behavior is same as config false.
+Gluten ignores `spark.sql.parquet.datetimeRebaseModeInRead`, it only returns what write in parquet file. It does not consider the difference between legacy
+hybrid (Julian Gregorian) calendar and Proleptic Gregorian calendar. The result may be different with vanilla spark.
+
+#### Parquet write conf
+Spark has `spark.sql.parquet.datetimeRebaseModeInWrite` config to decide whether legacy hybrid (Julian + Gregorian) calendar
+or Proleptic Gregorian calendar should be used during parquet writing for dates/timestamps. If the parquet to read is written
+by Spark with this config as true, Velox's TableScan will output different result when reading it back.
+
+#### Partition write (For Spark3.2 and Spark3.3)
+
+Gluten only supports static partition writes and does not support dynamic partition writes.
+
+```scala
+spark.sql("CREATE TABLE t (c int, d long, e long) STORED AS PARQUET partitioned by (c, d)")
+spark.sql("INSERT OVERWRITE TABLE t partition(c=1, d=2) SELECT 3 as e")
+```
+Gluten does not support dynamic partition write and bucket write, Exception may be raised if you use. e.g.,
+
+```scala
+spark.range(100).selectExpr("id as c1", "id % 7 as p")
+ .write
+ .format("parquet")
+ .partitionBy("p")
+ .save(f.getCanonicalPath)
+```
+
+#### Partition write (For Spark3.4 and later)
+
+Gluten supports static partition writes and dynamic partition writes.
+
+```scala
+spark.sql("CREATE TABLE t (c int, d long, e long) STORED AS PARQUET partitioned by (c, d)")
+spark.sql("INSERT OVERWRITE TABLE t partition(c=1, d) SELECT 2 as d, 3 as e")
+```
+
+Gluten does not support bucket write, and will fall back to vanilla Spark.
+
+```scala
+spark.range(100).selectExpr("id as c1", "id % 7 as p")
+ .write
+ .format("parquet")
+ .bucketBy(2, "c1")
+ .save(f.getCanonicalPath)
+```
+
+#### CTAS write (For Spark3.2 and Spark3.3)
+
+Gluten does not create table as select. It may raise exception. e.g.,
+
+```scala
+spark.range(100).toDF("id")
+ .write
+ .format("parquet")
+ .saveAsTable("velox_ctas")
+```
+
+#### CTAS write (For Spark3.4 and later)
+
+Gluten supports create table as select with parquet file format.
+
+```scala
+spark.range(100).toDF("id")
+ .write
+ .format("parquet")
+ .saveAsTable("velox_ctas")
+```
+
+#### HiveFileFormat write
+
+Gluten supports writes of HiveFileFormat when the output file type is of type `parquet` only
+
+#### NaN support
+Velox does NOT support NaN. So unexpected result can be obtained for a few cases, e.g., comparing a number with NaN.
+
+#### Configuration
+
+Parquet write only support three configs, other will not take effect.
+
+- compression code:
+ - sql conf: `spark.sql.parquet.compression.codec`
+ - option: `compression.codec`
+- block size
+ - sql conf: `spark.gluten.sql.columnar.parquet.write.blockSize`
+ - option: `parquet.block.size`
+- block rows
+ - sql conf: `spark.gluten.sql.native.parquet.write.blockRows`
+ - option: `parquet.block.rows`
+
+
+
+### Fetal error caused by Spark's columnar reading
+If the user enables Spark's columnar reading, error can occur due to Spark's columnar vector is not compatible with
+Gluten's.
+
+### Spill
+
+`OutOfMemoryException` may still be triggered within current implementation of spill-to-disk feature, when shuffle partitions is set to a large number. When this case happens, please try to reduce the partition number to get rid of the OOM.
+
+### Unsupported Data type support in ParquetScan
+
+- Byte type causes fallback to vanilla spark
+- Timestamp type
+
+ Only reading with INT96 and dictionary encoding is supported. When reading INT64 represented millisecond/microsecond timestamps, or INT96 represented timestamps of other encodings, exceptions can occur.
+
+- Complex types
+ - Parquet scan of nested array with struct or array as element type is not supported in Velox (fallback behavior).
+ - Parquet scan of nested map with struct as key type, or array type as value type is not supported in Velox (fallback behavior).
+
+### CSV Read
+The header option should be true. And now we only support DatasourceV1, i.e., user should set `spark.sql.sources.useV1SourceList=csv`. User defined read option is not supported, which will make CSV read fall back to vanilla Spark in most case.
+CSV read will also fall back to vanilla Spark and log warning when user specifies schema is different with file schema.
+
+### Utilizing Map Type as Hash Keys in ColumnarShuffleExchange
+Spark uses the `spark.sql.legacy.allowHashOnMapType` configuration to support hash map key functions.
+Gluten enables this configuration during the creation of ColumnarShuffleExchange, as shown in the code [link](https://github.com/apache/incubator-gluten/blob/0dacac84d3bf3d2759a5dd7e0735147852d2845d/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L355-L363).
+This method bypasses Spark's unresolved checks and creates projects with the hash(mapType) operator before ColumnarShuffleExchange.
+However, if `spark.sql.legacy.allowHashOnMapType` is disabled in a test environment, projects using the hash(mapType) expression may throw an `Invalid call to dataType on unresolved object` exception during validation, causing them to fallback to vanilla Spark, as referenced in the code [link](https://github.com/apache/spark/blob/de5fa426e23b84fc3c2bddeabcd2e1eda515abd5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L291-L296).
+ Enabling this configuration allows the project to be offloaded to Velox.
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-scalar-function-support.md b/_docs/v1.5.0/velox-backend/velox-backend-scalar-function-support.md
new file mode 100644
index 0000000..5fe0169
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-scalar-function-support.md
@@ -0,0 +1,458 @@
+---
+layout: page
+title: velox backend scalar function support
+nav_order: 4
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+# Scalar Functions Support Status
+
+**Out of 357 scalar functions in Spark 3.5, Gluten currently fully supports 239 functions and partially supports 24 functions.**
+
+## Array Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| array | CreateArray | S | |
+| array_append | ArrayAppend | S | |
+| array_compact | ArrayCompact | S | |
+| array_contains | ArrayContains | S | |
+| array_distinct | ArrayDistinct | S | |
+| array_except | ArrayExcept | S | |
+| array_insert | ArrayInsert | S | |
+| array_intersect | ArrayIntersect | S | |
+| array_join | ArrayJoin | S | |
+| array_max | ArrayMax | S | |
+| array_min | ArrayMin | S | |
+| array_position | ArrayPosition | S | |
+| array_prepend | ArrayPrepend | S | |
+| array_remove | ArrayRemove | S | |
+| array_repeat | ArrayRepeat | S | |
+| array_union | ArrayUnion | S | |
+| arrays_overlap | ArraysOverlap | S | |
+| arrays_zip | ArraysZip | S | |
+| flatten | Flatten | S | |
+| get | Get | S | |
+| sequence | Sequence | | |
+| shuffle | Shuffle | S | |
+| slice | Slice | S | |
+| sort_array | SortArray | S | |
+
+## Bitwise Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|--------------------|---------------------|----------|----------------|
+| & | BitwiseAnd | S | |
+| ^ | BitwiseXor | S | |
+| bit_count | BitwiseCount | S | |
+| bit_get | BitwiseGet | S | |
+| getbit | BitwiseGet | S | |
+| shiftright | ShiftRight | S | |
+| shiftrightunsigned | ShiftRightUnsigned | | |
+| | | BitwiseOr | S | |
+| ~ | BitwiseNot | S | |
+
+## Collection Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| array_size | ArraySize | S | |
+| cardinality | Size | S | |
+| concat | Concat | S | |
+| reverse | Reverse | S | |
+| size | Size | S | |
+
+## Conditional Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| coalesce | Coalesce | S | |
+| if | If | S | |
+| ifnull | Nvl | S | |
+| nanvl | NaNvl | S | |
+| nullif | NullIf | S | |
+| nvl | Nvl | S | |
+| nvl2 | Nvl2 | S | |
+| when | CaseWhen | S | |
+
+## Conversion Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| bigint | | S | |
+| binary | | S | |
+| boolean | | S | |
+| cast | Cast | S | |
+| date | | S | |
+| decimal | | S | |
+| double | | S | |
+| float | | S | |
+| int | | S | |
+| smallint | | S | |
+| string | | S | |
+| timestamp | | S | |
+| tinyint | | S | |
+
+## Csv Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| from_csv | CsvToStructs | | |
+| schema_of_csv | SchemaOfCsv | | |
+| to_csv | StructsToCsv | | |
+
+## Date and Timestamp Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|---------------------|--------------------------------------|----------|----------------|
+| add_months | AddMonths | S | |
+| convert_timezone | ConvertTimezone | | |
+| curdate | CurDateExpressionBuilder | | |
+| current_date | CurrentDate | | |
+| current_timestamp | CurrentTimestamp | | |
+| current_timezone | CurrentTimeZone | | |
+| date_add | DateAdd | S | |
+| date_diff | DateDiff | S | |
+| date_format | DateFormatClass | S | |
+| date_from_unix_date | DateFromUnixDate | S | |
+| date_part | DatePartExpressionBuilder | | |
+| date_sub | DateSub | S | |
+| date_trunc | TruncTimestamp | S | |
+| dateadd | DateAdd | S | |
+| datediff | DateDiff | S | |
+| datepart | DatePartExpressionBuilder | | |
+| day | DayOfMonth | S | |
+| dayofmonth | DayOfMonth | S | |
+| dayofweek | DayOfWeek | S | |
+| dayofyear | DayOfYear | S | |
+| extract | Extract | S | |
+| from_unixtime | FromUnixTime | S | |
+| from_utc_timestamp | FromUTCTimestamp | S | |
+| hour | Hour | S | |
+| last_day | LastDay | S | |
+| localtimestamp | LocalTimestamp | | |
+| make_date | MakeDate | S | |
+| make_dt_interval | MakeDTInterval | | |
+| make_interval | MakeInterval | | |
+| make_timestamp | MakeTimestamp | S | |
+| make_timestamp_ltz | MakeTimestampLTZExpressionBuilder | | |
+| make_timestamp_ntz | MakeTimestampNTZExpressionBuilder | | |
+| make_ym_interval | MakeYMInterval | S | |
+| minute | Minute | S | |
+| month | Month | S | |
+| months_between | MonthsBetween | | |
+| next_day | NextDay | S | |
+| now | Now | | |
+| quarter | Quarter | S | |
+| second | Second | S | |
+| session_window | SessionWindow | | |
+| timestamp_micros | MicrosToTimestamp | S | |
+| timestamp_millis | MillisToTimestamp | S | |
+| timestamp_seconds | SecondsToTimestamp | | |
+| to_date | ParseToDate | | |
+| to_timestamp | ParseToTimestamp | | |
+| to_timestamp_ltz | ParseToTimestampLTZExpressionBuilder | | |
+| to_timestamp_ntz | ParseToTimestampNTZExpressionBuilder | | |
+| to_unix_timestamp | ToUnixTimestamp | PS | |
+| to_utc_timestamp | ToUTCTimestamp | S | |
+| trunc | TruncDate | S | |
+| try_to_timestamp | TryToTimestampExpressionBuilder | | |
+| unix_date | UnixDate | S | |
+| unix_micros | UnixMicros | S | |
+| unix_millis | UnixMillis | S | |
+| unix_seconds | UnixSeconds | S | |
+| unix_timestamp | UnixTimestamp | S | |
+| weekday | WeekDay | S | |
+| weekofyear | WeekOfYear | S | |
+| window | TimeWindow | | |
+| window_time | WindowTime | | |
+| year | Year | S | |
+
+## Hash Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| crc32 | Crc32 | S | |
+| hash | Murmur3Hash | S | |
+| md5 | Md5 | S | |
+| sha | Sha1 | S | |
+| sha1 | Sha1 | S | |
+| sha2 | Sha2 | S | |
+| xxhash64 | XxHash64 | S | |
+
+## JSON Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| from_json | JsonToStructs | PS | from_json with 'spark.sql.caseSensitive = true' is not supported in Velox<br>from_json with 'spark.sql.json.enablePartialResults = false' is not supported in Velox<br>from_json with column corrupt record is not supported in Velox<br>from_json with duplicate keys is not supported in Velox<br>from_json with options is not supported in Velox |
+| get_json_object | GetJsonObject | S | |
+| json_array_length | LengthOfJsonArray | S | |
+| json_object_keys | JsonObjectKeys | S | |
+| json_tuple | JsonTuple | S | |
+| schema_of_json | SchemaOfJson | | |
+| to_json | StructsToJson | PS | to_json with options is not supported in Velox |
+
+## Lambda Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| aggregate | ArrayAggregate | S | |
+| array_sort | ArraySort | S | |
+| exists | ArrayExists | S | |
+| filter | ArrayFilter | S | |
+| forall | ArrayForAll | S | |
+| map_filter | MapFilter | S | |
+| map_zip_with | MapZipWith | S | |
+| reduce | ArrayAggregate | S | |
+| transform | ArrayTransform | S | |
+| transform_keys | TransformKeys | S | |
+| transform_values | TransformValues | S | |
+| zip_with | ZipWith | S | |
+
+## Map Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|-----------------------------------------------------------------------------|
+| element_at | ElementAt | S | |
+| map | CreateMap | PS | |
+| map_concat | MapConcat | PS | |
+| map_contains_key | MapContainsKey | S | |
+| map_entries | MapEntries | S | |
+| map_from_arrays | MapFromArrays | | |
+| map_from_entries | MapFromEntries | | |
+| map_keys | MapKeys | S | |
+| map_values | MapValues | S | |
+| str_to_map | StringToMap | PS | Only spark.sql.mapKeyDedupPolicy = EXCEPTION is supported for Velox backend |
+| try_element_at | TryElementAt | | |
+
+## Mathematical Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|------------------------|----------|----------------|
+| % | Remainder | S | |
+| * | Multiply | S | |
+| + | Add | S | |
+| - | Subtract | S | |
+| / | Divide | S | |
+| abs | Abs | S | |
+| acos | Acos | S | |
+| acosh | Acosh | S | |
+| asin | Asin | S | |
+| asinh | Asinh | S | |
+| atan | Atan | S | |
+| atan2 | Atan2 | S | |
+| atanh | Atanh | S | |
+| bin | Bin | S | |
+| bround | BRound | | |
+| cbrt | Cbrt | S | |
+| ceil | CeilExpressionBuilder | PS | |
+| ceiling | CeilExpressionBuilder | PS | |
+| conv | Conv | S | |
+| cos | Cos | S | |
+| cosh | Cosh | S | |
+| cot | Cot | S | |
+| csc | Csc | S | |
+| degrees | ToDegrees | S | |
+| div | IntegralDivide | | |
+| e | EulerNumber | S | |
+| exp | Exp | S | |
+| expm1 | Expm1 | S | |
+| factorial | Factorial | S | |
+| floor | FloorExpressionBuilder | PS | |
+| greatest | Greatest | S | |
+| hex | Hex | S | |
+| hypot | Hypot | S | |
+| least | Least | S | |
+| ln | Log | | |
+| log | Logarithm | S | |
+| log10 | Log10 | S | |
+| log1p | Log1p | S | |
+| log2 | Log2 | S | |
+| mod | Remainder | S | |
+| negative | UnaryMinus | S | |
+| pi | Pi | S | |
+| pmod | Pmod | S | |
+| positive | UnaryPositive | S | |
+| pow | Pow | S | |
+| power | Pow | S | |
+| radians | ToRadians | | |
+| rand | Rand | S | |
+| randn | Randn | | |
+| random | Rand | S | |
+| rint | Rint | S | |
+| round | Round | S | |
+| sec | Sec | S | |
+| shiftleft | ShiftLeft | S | |
+| sign | Signum | S | |
+| signum | Signum | S | |
+| sin | Sin | | |
+| sinh | Sinh | S | |
+| sqrt | Sqrt | S | |
+| tan | Tan | | |
+| tanh | Tanh | | |
+| try_add | TryAdd | PS | |
+| try_divide | TryDivide | | |
+| try_multiply | TryMultiply | | |
+| try_subtract | TrySubtract | | |
+| unhex | Unhex | S | |
+| width_bucket | WidthBucket | S | |
+
+## Misc Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-----------------------------|---------------------------|----------|----------------|
+| aes_decrypt | AesDecrypt | | |
+| aes_encrypt | AesEncrypt | | |
+| assert_true | AssertTrue | S | |
+| bitmap_bit_position | BitmapBitPosition | | |
+| bitmap_bucket_number | BitmapBucketNumber | | |
+| bitmap_count | BitmapCount | | |
+| current_catalog | CurrentCatalog | | |
+| current_database | CurrentDatabase | | |
+| current_schema | CurrentDatabase | | |
+| current_user | CurrentUser | | |
+| equal_null | EqualNull | S | |
+| hll_sketch_estimate | HllSketchEstimate | | |
+| hll_union | HllUnion | | |
+| input_file_block_length | InputFileBlockLength | | |
+| input_file_block_start | InputFileBlockStart | | |
+| input_file_name | InputFileName | | |
+| java_method | CallMethodViaReflection | | |
+| monotonically_increasing_id | MonotonicallyIncreasingID | | |
+| reflect | CallMethodViaReflection | | |
+| spark_partition_id | SparkPartitionID | S | |
+| try_aes_decrypt | TryAesDecrypt | | |
+| typeof | TypeOf | | |
+| user | CurrentUser | | |
+| uuid | Uuid | S | |
+| version | SparkVersion | S | |
+| || | | S | |
+
+## Predicate Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|------------------------|
+| ! | Not | S | |
+| != | | S | |
+| < | LessThan | S | |
+| <= | LessThanOrEqual | S | |
+| <=> | EqualNullSafe | S | |
+| <> | | S | |
+| = | EqualTo | S | |
+| == | EqualTo | S | |
+| > | GreaterThan | S | |
+| >= | GreaterThanOrEqual | S | |
+| and | And | S | |
+| between | | S | |
+| case | | S | |
+| ilike | ILike | S | |
+| in | In | PS | |
+| isnan | IsNaN | S | |
+| isnotnull | IsNotNull | S | |
+| isnull | IsNull | S | |
+| like | Like | S | |
+| not | Not | S | |
+| or | Or | S | |
+| regexp | RLike | PS | Lookaround unsupported |
+| regexp_like | RLike | PS | Lookaround unsupported |
+| rlike | RLike | PS | Lookaround unsupported |
+
+## String Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|--------------------|-----------------------------|----------|---------------------------------------------------------|
+| ascii | Ascii | S | |
+| base64 | Base64 | PS | base64 with chunkBase64String disabled is not supported |
+| bit_length | BitLength | S | |
+| btrim | StringTrimBoth | S | |
+| char | Chr | S | |
+| char_length | Length | S | |
+| character_length | Length | S | |
+| chr | Chr | S | |
+| concat_ws | ConcatWs | S | |
+| contains | ContainsExpressionBuilder | PS | BinaryType unsupported |
+| decode | Decode | | |
+| elt | Elt | | |
+| encode | Encode | | |
+| endswith | EndsWithExpressionBuilder | PS | BinaryType unsupported |
+| find_in_set | FindInSet | S | |
+| format_number | FormatNumber | | |
+| format_string | FormatString | | |
+| initcap | InitCap | S | |
+| instr | StringInstr | S | |
+| lcase | Lower | S | |
+| left | Left | S | |
+| len | Length | S | |
+| length | Length | S | |
+| levenshtein | Levenshtein | S | |
+| locate | StringLocate | S | |
+| lower | Lower | S | |
+| lpad | LPadExpressionBuilder | PS | BinaryType unsupported |
+| ltrim | StringTrimLeft | S | |
+| luhn_check | Luhncheck | S | |
+| mask | MaskExpressionBuilder | S | |
+| octet_length | OctetLength | | |
+| overlay | Overlay | S | |
+| position | StringLocate | S | |
+| printf | FormatString | | |
+| regexp_count | RegExpCount | | |
+| regexp_extract | RegExpExtract | PS | Lookaround unsupported |
+| regexp_extract_all | RegExpExtractAll | PS | Lookaround unsupported |
+| regexp_instr | RegExpInStr | | |
+| regexp_replace | RegExpReplace | PS | Lookaround unsupported |
+| regexp_substr | RegExpSubStr | | |
+| repeat | StringRepeat | S | |
+| replace | StringReplace | S | |
+| right | Right | S | |
+| rpad | RPadExpressionBuilder | PS | BinaryType unsupported |
+| rtrim | StringTrimRight | S | |
+| sentences | Sentences | | |
+| soundex | SoundEx | S | |
+| space | StringSpace | | |
+| split | StringSplit | S | |
+| split_part | SplitPart | S | |
+| startswith | StartsWithExpressionBuilder | PS | BinaryType unsupported |
+| substr | Substring | PS | |
+| substring | Substring | PS | |
+| substring_index | SubstringIndex | S | |
+| to_binary | ToBinary | | |
+| to_char | ToCharacter | | |
+| to_number | ToNumber | | |
+| to_varchar | ToCharacter | | |
+| translate | StringTranslate | S | |
+| trim | StringTrim | S | |
+| try_to_binary | TryToBinary | | |
+| try_to_number | TryToNumber | | |
+| ucase | Upper | S | |
+| unbase64 | UnBase64 | PS | unbase64 with failOnError is not supported |
+| upper | Upper | S | |
+
+## Struct Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| named_struct | CreateNamedStruct | S | |
+| struct | | S | |
+
+## URL Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| parse_url | ParseUrl | | |
+| url_decode | UrlDecode | S | |
+| url_encode | UrlEncode | S | |
+
+## XML Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| xpath | XPathList | | |
+| xpath_boolean | XPathBoolean | | |
+| xpath_double | XPathDouble | | |
+| xpath_float | XPathFloat | | |
+| xpath_int | XPathInt | | |
+| xpath_long | XPathLong | | |
+| xpath_number | XPathDouble | | |
+| xpath_short | XPathShort | | |
+| xpath_string | XPathString | | |
+
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-support-progress.md b/_docs/v1.5.0/velox-backend/velox-backend-support-progress.md
new file mode 100644
index 0000000..9e9c02c
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-support-progress.md
@@ -0,0 +1,125 @@
+---
+layout: page
+title: velox backend support progress
+nav_order: 5
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+# The Operators and Functions Support Progress
+
+Gluten is still under active development. Here is a list of supported operators and functions.
+
+Since the same function may have different semantics between Presto and Spark, Velox implement the functions in Presto category, if we note a
+different semantics from Spark, then the function is implemented in Spark category. So Gluten will first try to find function in Velox's spark
+category, if a function isn't implemented then refer to Presto category.
+
+We use some notations to describe the supporting status of operators/functions in the tables below, they are:
+
+| Value | Description |
+|--------------|------------------------------------------------------------------------------------------|
+| S | Supported. Gluten or Velox supports fully. |
+| S* | Mark for foldable expression that will be converted to alias after spark's optimization. |
+| [Blank Cell] | Not applicable case or needs to confirm. |
+| PS | Partial Support. Velox only partially supports it. |
+| NS | Not Supported. Velox backend does not support it. |
+
+And also some notations for the function implementation's restrictions:
+
+| Value | Description |
+|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mismatched | Some functions are implemented by Velox, but have different semantics from Apache Spark, we mark them as "Mismatched". |
+| ANSI OFF | Gluten doesn't support [ANSI mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html). If it is enabled, Gluten will fall back to Vanilla Spark. |
+
+## Operator Map
+
+Gluten supports 30+ operators (Drag to right to see all data types)
+
+| Executor | Description | Gluten Name | Velox Name | BOOLEAN | BYTE | SHORT | INT | LONG | FLOAT | DOUBLE | STRING | NULL | BINARY | ARRAY | MAP | STRUCT(ROW) | DATE | TIMESTAMP | DECIMAL | CALENDAR | UDT |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------|---------|------|-------|-----|------|-------|--------|--------|------|--------|-------|-----|-------------|------|-----------|---------|----------|-----|
+| FileSourceScanExec | Reading data from files, often from Hive tables | FileSourceScanExecTransformer | TableScanNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| BatchScanExec | The backend for most file input | BatchScanExecTransformer | TableScanNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| FilterExec | The backend for most filter statements | FilterExecTransformer | FilterNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ProjectExec | The backend for most select, withColumn and dropColumn statements | ProjectExecTransformer | ProjectNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| HashAggregateExec | The backend for hash based aggregations | HashAggregateBaseTransformer | AggregationNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| BroadcastHashJoinExec | Implementation of join using broadcast data | BroadcastHashJoinExecTransformer | HashJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ShuffledHashJoinExec | Implementation of join using hashed shuffled data | ShuffleHashJoinExecTransformer | HashJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| SortExec | The backend for the sort operator | SortExecTransformer | OrderByNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| SortMergeJoinExec | Sort merge join, replacing with shuffled hash join | SortMergeJoinExecTransformer | MergeJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| WindowExec | Window operator backend | WindowExecTransformer | WindowNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| GlobalLimitExec | Limiting of results across partitions | LimitTransformer | LimitNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| LocalLimitExec | Per-partition limiting of results | LimitTransformer | LimitNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ExpandExec | The backend for the expand operator | ExpandExecTransformer | GroupIdNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| UnionExec | The backend for the union operator | UnionExecTransformer | N | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| DataWritingCommandExec | Writing data | Y | TableWriteNode | S | S | S | S | S | S | S | S | S | S | S | NS | S | S | NS | S | NS | NS |
+| CartesianProductExec | Implementation of join using brute force | CartesianProductExecTransformer | NestedLoopJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ShuffleExchangeExec | The backend for most data being exchanged between processes | ColumnarShuffleExchangeExec | ExchangeNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The unnest operation expands arrays and maps into separate columns | N | UnnestNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The top-n operation reorders a dataset based on one or more identified sort fields as well as a sorting order | N | TopNNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The partitioned output operation redistributes data based on zero or more distribution fields | N | PartitionedOutputNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The values operation returns specified data | N | ValuesNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | A receiving operation that merges multiple ordered streams to maintain orderedness | N | MergeExchangeNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | An operation that merges multiple ordered streams to maintain orderedness | N | LocalMergeNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | Partitions input data into multiple streams or combines data from multiple streams into a single stream | N | LocalPartitionNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The enforce single row operation checks that input contains at most one row and returns that row unmodified | N | EnforceSingleRowNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The assign unique id operation adds one column at the end of the input columns with unique value per row | N | AssignUniqueIdNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | S | S | S | S | S |
+| ReusedExchangeExec | A wrapper for reused exchange to have different output | ReusedExchangeExec | N | | | | | | | | | | | | | | | | | | |
+| CollectLimitExec | Reduce to single partition and apply limit | ColumnarCollectLimitExec | N | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S |
+| CollectTailExec | Collect the tail `x` elements from dataframe | ColumnarCollectTailExec | N | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S |
+| BroadcastExchangeExec | The backend for broadcast exchange of data | Y | Y | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | S | NS | NS |
+| ObjectHashAggregateExec | The backend for hash based aggregations supporting TypedImperativeAggregate functions | HashAggregateExecBaseTransformer | N | | | | | | | | | | | | | | | | | | |
+| SortAggregateExec | The backend for sort based aggregations | HashAggregateExecBaseTransformer (Partially supported) | N | | | | | | | | | | | | | | | | | | |
+| CoalesceExec | Reduce the partition numbers | CoalesceExecTransformer | N | | | | | | | | | | | | | | | | | | |
+| GenerateExec | The backend for operations that generate more output rows than input rows like explode | GenerateExecTransformer | UnnestNode | | | | | | | | | | | | | | | | | | |
+| RangeExec | The backend for range operator | ColumnarRangeExec | N | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S |
+| SampleExec | The backend for the sample operator | SampleExecTransformer | N | | | | | | | | | | | | | | | | | | |
+| SubqueryBroadcastExec | Plan to collect and transform the broadcast key values | Y | Y | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | S | NS | NS |
+| TakeOrderedAndProjectExec | Take the first limit elements as defined by the sortOrder, and do projection if needed | Y | Y | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | S | NS | NS |
+| CustomShuffleReaderExec | A wrapper of shuffle query stage | N | N | | | | | | | | | | | | | | | | | | |
+| InMemoryTableScanExec | Implementation of InMemory Table Scan | Y | Y | | | | | | | | | | | | | | | | | | |
+| BroadcastNestedLoopJoinExec | Implementation of join using brute force. Full outer joins and joins where the broadcast side matches the join side (e.g.: LeftOuter with left broadcast) are not supported | BroadcastNestedLoopJoinExecTransformer | NestedLoopJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| AggregateInPandasExec | The backend for an Aggregation Pandas UDF, this accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| ArrowEvalPythonExec | The backend of the Scalar Pandas UDFs. Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| FlatMapGroupsInPandasExec | The backend for Flat Map Groups Pandas UDF, Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| MapInPandasExec | The backend for Map Pandas Iterator UDF. Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| WindowInPandasExec | The backend for Window Aggregation Pandas UDF, Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| HiveTableScanExec | The Hive table scan operator. Column and partition pruning are both handled | Y | Y | | | | | | | | | | | | | | | | | | |
+| InsertIntoHiveTable | Command for writing data out to a Hive table | Y | Y | | | | | | | | | | | | | | | | | | |
+| Velox2Row | Convert Velox format to Row format | Y | Y | S | S | S | S | S | S | S | S | NS | S | NS | NS | NS | S | S | NS | NS | NS |
+| Velox2Arrow | Convert Velox format to Arrow format | Y | Y | S | S | S | S | S | S | S | S | NS | S | S | S | S | S | NS | S | NS | NS |
+| WindowGroupLimitExec | Optimize window with rank like function with filter on it | Y | Y | S | S | S | S | S | S | S | S | NS | S | S | S | S | S | NS | S | NS | NS |
+
+## Function Support Status
+
+Spark categorizes built-in functions into four types: Scalar Functions, Aggregate Functions,
+Window Functions, and Generator Functions.
+In Gluten, function support is automatically generated by a script and maintained in separate files.
+
+When running the script, the `--spark_home` arg should be set to either:
+* The directory containing the Spark source code for the latest supported Spark version in Gluten, and the Spark
+ project must be built from source.
+* Or use the `install_spark_resources.sh` script to get a directory with the necessary resource files:
+ ```
+ # Define a directory to use for the Spark files and the latest Spark version
+ export spark_dir=/tmp/spark
+ export spark_version=3.5
+
+ # Run the install_spark_resources.sh script
+ .github/workflows/util/install_spark_resources.sh ${spark_version} ${spark_dir}
+ ```
+ After running the `install_spark_resources.sh`, the `--spark_home` for the document generation script will be
+ something like: `--spark_home=${spark_dir}/shims/spark35/spark_home"`
+
+Use the following command to generate and update the support status:
+```shell
+python3 tools/scripts/gen-function-support-docs.py --spark_home=/path/to/spark_source_code
+```
+
+Please check the links below for the detailed support status of each category:
+
+[Scalar Functions Support Status](./velox-backend-scalar-function-support.md)
+
+[Aggregate Functions Support Status](./velox-backend-aggregate-function-support.md)
+
+[Window Functions Support Status](./velox-backend-window-function-support.md)
+
+[Generator Functions Support Status](./velox-backend-generator-function-support.md)
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-troubleshooting.md b/_docs/v1.5.0/velox-backend/velox-backend-troubleshooting.md
new file mode 100644
index 0000000..c66f619
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-troubleshooting.md
@@ -0,0 +1,99 @@
+---
+layout: page
+title: velox backend troubleshooting
+nav_order: 6
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+## Troubleshooting
+
+### Fatal error after native exception is thrown
+
+We depend on checking exceptions thrown from native code to validate whether a spark plan can
+be really offloaded to native engine. But if `libunwind-dev` is installed, native exception will not be
+caught and will interrupt the program. So far, we observed this fatal error can happen only on Ubuntu 20.04.
+Please remove `libunwind-dev` and then re-build the project to address this issue.
+
+`sudo apt-get purge --auto-remove libunwind-dev`
+
+### Jar conflict issue
+
+With the latest version of Gluten, there should not be any jar conflict issue anymore. If you still get hit with
+such issue, please follow the below instructions.
+
+The potentially conflicting libraries include protobuf (Both Velox and CK backend), flatbuffers (Velox backend), and arrow-* (Velox backend).
+These libraries are compiled from source and packed into Gluten's jar. Jvm should search them from Gluten.jar firstly and load them. But for
+some reason jvm loads the jars from spark_home/jars which causes conflict. You may use below commands to remove the jars from spark_home/jars.
+We are still investigating the root cause. Welcome to share if you have good solution.
+
+```
+rm -rf $SPARK_HOME/jars/protobuf-*
+# velox backend only
+rm -rf $SPARK_HOME/jars/flatbuffers-*
+rm -rf $SPARK_HOME/jars/arrow-*
+```
+
+### Incompatible class error when using native writer
+Gluten native writer overwrite some vanilla spark classes. Therefore, when running a program that uses gluten, it is essential to ensure that
+the gluten jar is loaded prior to the vanilla spark jar. In this section, we will provide some configuration settings in
+`$SPARK_HOME/conf/spark-defaults.conf` for Yarn client, Yarn cluster, and Local&Standalone mode to guarantee that the gluten jar is prioritized.
+
+#### Configurations for Yarn Client mode
+
+```
+// spark will upload the gluten jar to hdfs and then the nodemanager will fetch the gluten jar before start the executor process. Here also can set the spark.jars.
+spark.files=<absolute_path_to_Gluten_JAR>
+// The absolute path on running node
+spark.driver.extraClassPath=<absolute_path_to_Gluten_JAR>
+// The relative path under the executor working directory
+spark.executor.extraClassPath=./<Gluten_JAR>
+```
+
+#### Configurations for Yarn Cluster mode
+```
+spark.driver.userClassPathFirst=true
+spark.executor.userClassPathFirst=true
+
+spark.files=<absolute_path_to_Gluten_JAR>
+// The relative path under the executor working directory
+spark.driver.extraClassPath=./<Gluten_JAR>
+// The relative path under the executor working directory
+spark.executor.extraClassPath=./<Gluten_JAR>
+```
+#### Configurations for Local & Standalone mode
+```
+// The absolute path on running node
+spark.driver.extraClassPath=<absolute_path_to_Gluten_JAR>
+// The absolute path on running node
+spark.executor.extraClassPath=<absolute_path_to_Gluten_JAR>
+```
+
+### Invalid pointer error
+
+If the below error is reported at runtime, please re-build gluten with `--compile_arrow_java=ON`, then redeploy Gluten jar.
+
+```
+*** Error in `/usr/local/jdk1.8.0_381/bin/java': free(): invalid pointer: 0x00007f36cb5cec80 ***
+======= Backtrace: =========
+/lib64/libc.so.6(+0x7d1fd)[0x7f38c29da1fd]
+/lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7f36cb3370d2]
+/lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7f36cb337523]
+/lib64/libstdc++.so.6(+0x71495)[0x7f36cb338495]
+/lib64/libpthread.so.0(pthread_once+0x50)[0x7f38c3147be0]
+/lib64/libstdc++.so.6(+0x714e1)[0x7f36cb3384e1]
+/lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7f36cb338523]
+/lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7f36cb33537c]
+/tmp/jnilib-645156599284574767.tmp(+0x2a90)[0x7f375d235a90]
+/lib64/ld-linux-x86-64.so.2(+0xf4e3)[0x7f38c33664e3]
+/lib64/ld-linux-x86-64.so.2(+0x13b04)[0x7f38c336ab04]
+/lib64/ld-linux-x86-64.so.2(+0xf2f4)[0x7f38c33662f4]
+/lib64/ld-linux-x86-64.so.2(+0x1321b)[0x7f38c336a21b]
+/lib64/libdl.so.2(+0x102b)[0x7f38c2d1f02b]
+/lib64/ld-linux-x86-64.so.2(+0xf2f4)[0x7f38c33662f4]
+/lib64/libdl.so.2(+0x162d)[0x7f38c2d1f62d]
+/lib64/libdl.so.2(dlopen+0x31)[0x7f38c2d1f0c1]
+/usr/local/jdk1.8.0_381/jre/lib/amd64/server/libjvm.so(+0x9292b1)[0x7f38c22732b1]
+/usr/local/jdk1.8.0_381/jre/lib/amd64/server/libjvm.so(JVM_LoadLibrary+0xa1)[0x7f38c205e0c1]
+/usr/local/jdk1.8.0_381/jre/lib/amd64/libjava.so(Java_java_lang_ClassLoader_00024NativeLibrary_load+0x1ac)
+...
+```
diff --git a/_docs/v1.5.0/velox-backend/velox-backend-window-function-support.md b/_docs/v1.5.0/velox-backend/velox-backend-window-function-support.md
new file mode 100644
index 0000000..b161db8
--- /dev/null
+++ b/_docs/v1.5.0/velox-backend/velox-backend-window-function-support.md
@@ -0,0 +1,25 @@
+---
+layout: page
+title: velox backend window function support
+nav_order: 7
+parent: Velox Backend
+grand_parent: v1.5.0
+---
+# Window Functions Support Status
+
+**Out of 9 window functions in Spark 3.5, Gluten currently fully supports 9 functions.**
+
+## Window Functions
+
+| Spark Functions | Spark Expressions | Status | Restrictions |
+|-------------------|---------------------|----------|----------------|
+| cume_dist | CumeDist | S | |
+| dense_rank | DenseRank | S | |
+| lag | Lag | S | |
+| lead | Lead | S | |
+| nth_value | NthValue | S | |
+| ntile | NTile | S | |
+| percent_rank | PercentRank | S | |
+| rank | Rank | S | |
+| row_number | RowNumber | S | |
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org