You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by we...@apache.org on 2024/08/08 09:51:21 UTC
(incubator-gluten-site) branch main updated: Update documentations to v1.2.0 (#20)
This is an automated email from the ASF dual-hosted git repository.
weitingchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten-site.git
The following commit(s) were added to refs/heads/main by this push:
new 41f97f8 Update documentations to v1.2.0 (#20)
41f97f8 is described below
commit 41f97f8c1727555cf7322cdf0fc7c60c0afdf9ef
Author: Wei-Ting Chen <we...@apache.org>
AuthorDate: Thu Aug 8 17:51:17 2024 +0800
Update documentations to v1.2.0 (#20)
---
_config.yml | 17 +-
archives/index.md | 2 +-
docs/ClickHouse.md | 9 -
docs/Configuration.md | 112 +++++
docs/Velox.md | 9 -
docs/VeloxBuildGuide.md | 78 ----
{developers => docs/developers}/CppCodingStyle.md | 2 +
{developers => docs/developers}/HowTo.md | 4 +-
{developers => docs/developers}/HowToRelease.md | 0
{developers => docs/developers}/MicroBenchmarks.md | 2 +
{developers => docs/developers}/NewToGluten.md | 4 +-
.../developers}/SubstraitModifications.md | 2 +
.../developers}/UsingGperftoolsInCH.md | 0
.../developers}/UsingJemallocWithCH.md | 0
{developers => docs/developers}/docker_centos7.md | 4 +-
{developers => docs/developers}/docker_centos8.md | 4 +-
.../developers}/docker_ubuntu22.04.md | 2 +
{developers => docs/developers}/index.md | 3 +-
.../velox-function-development-guide.md | 2 +
.../ClickHouse.md} | 70 ++--
.../Velox.md} | 346 ++++++----------
docs/get-started/VeloxABFS.md | 25 ++
docs/{ => get-started}/VeloxGCS.md | 22 +-
docs/get-started/VeloxLocalCache.md | 22 +
docs/{ => get-started}/VeloxS3.md | 22 +-
docs/get-started/Work-with-pyspark.ipynb | 136 ++++++
docs/get-started/build-guide.md | 81 ++++
docs/get-started/index.md | 11 +
docs/index.md | 2 +-
docs/velox-backend/index.md | 11 +
docs/velox-backend/velox-backend-limitations.md | 168 ++++++++
.../velox-backend-support-progress.md | 461 +++++++++++++++++++++
.../velox-backend/velox-backend-troubleshooting.md | 100 +++++
33 files changed, 1352 insertions(+), 381 deletions(-)
diff --git a/_config.yml b/_config.yml
index 9940568..59054db 100644
--- a/_config.yml
+++ b/_config.yml
@@ -93,7 +93,7 @@ search:
mermaid:
# Version of mermaid library
# Pick an available version from https://cdn.jsdelivr.net/npm/mermaid/
- version: "10.8.0"
+ version: "10.9.1"
# Aux links for the upper right navigation
aux_links:
@@ -101,7 +101,11 @@ aux_links:
- "//github.com/apache/incubator-gluten"
# Makes Aux links open in a new tab. Default is false
-aux_links_new_tab: false
+aux_links_new_tab: true
+
+# Enable or disable the side/mobile menu globally
+# Nav menu can also be selectively enabled or disabled using page variables or the minimal layout
+nav_enabled: true
# Heading anchor links appear on hover over h1-h6 tags in page content
# allowing users to deep link to a particular heading on a page.
@@ -109,7 +113,14 @@ aux_links_new_tab: false
# Supports true (default) or false
heading_anchors: true
-color_scheme: light
+# External navigation links
+#nav_external_links:
+# - title: Just the Docs on GitHub
+# url: https://github.com/just-the-docs/just-the-docs
+# hide_icon: false # set to true to hide the external link icon - defaults to false
+# opens_in_new_tab: false # set to true to open this link in a new tab - defaults to false
+
+color_scheme: dark
# Footer content
# appears at the bottom of every page's main content
diff --git a/archives/index.md b/archives/index.md
index 87d845d..3ddbdae 100644
--- a/archives/index.md
+++ b/archives/index.md
@@ -1,7 +1,7 @@
---
layout: page
title: Archives
-nav_order: 4
+nav_order: 3
has_children: true
permalink: /archives/
---
diff --git a/docs/ClickHouse.md b/docs/ClickHouse.md
deleted file mode 100644
index cfc1b16..0000000
--- a/docs/ClickHouse.md
+++ /dev/null
@@ -1,9 +0,0 @@
----
-layout: page
-title: ClickHouse Backend
-nav_order: 2
-has_children: true
-parent: Documentation
-permalink: /clickhouse/
----
-# Gluten Documents by version
diff --git a/docs/Configuration.md b/docs/Configuration.md
new file mode 100644
index 0000000..90b8d57
--- /dev/null
+++ b/docs/Configuration.md
@@ -0,0 +1,112 @@
+---
+layout: page
+title: Configuration
+nav_order: 2
+parent: Documentations
+permalink: /docs/configuration/
+---
+
+# Spark Configurations for Gluten Plugin
+
+There are many configurations could impact the Gluten Plugin performance and can be fine-tuned in Spark.
+You can add these configurations into spark-defaults.conf to enable or disable the setting.
+
+## Spark parameters
+
+| Parameters | Description [...]
+|------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| spark.driver.extraClassPath | To add Gluten Plugin jar file in Spark Driver [...]
+| spark.executor.extraClassPath | To add Gluten Plugin jar file in Spark Executor [...]
+| spark.executor.memory | To set up how much memory to be used for Spark Executor. [...]
+| spark.memory.offHeap.size | To set up how much memory to be used for Java OffHeap.<br /> Please notice Gluten Plugin will leverage this setting to allocate memory space for native usage even offHeap is disabled. <br /> The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Gluten Plugin [...]
+| spark.sql.sources.useV1SourceList | Choose to use V1 source [...]
+| spark.sql.join.preferSortMergeJoin | To turn off preferSortMergeJoin in Spark [...]
+| spark.plugins | To load Gluten's components by Spark's plug-in loader [...]
+| spark.shuffle.manager | To turn on Gluten Columnar Shuffle Plugin [...]
+| spark.gluten.enabled | Enable Gluten, default is true. Just an experimental property. Recommend to enable/disable Gluten through the setting for `spark.plugins`. [...]
+| spark.gluten.memory.isolation | (Experimental) Enable isolated memory mode. If true, Gluten controls the maximum off-heap memory can be used by each task to X, X = executor memory / max task slots. It's recommended to set true if Gluten serves concurrent queries within a single session, since not all memory Gluten allocated is guaranteed to be spillable. In the case, the feature should be enabled to avoid OOM. Note when true, setting spark.memory.storageFra [...]
+| spark.gluten.ras.enabled | Experimental: Enables RAS (relation algebra selector) during physical planning to generate more efficient query plan. Note, this feature is still in development and may not bring performance profits. [...]
+| spark.gluten.sql.columnar.maxBatchSize | Number of rows to be processed in each batch. Default value is 4096. [...]
+| spark.gluten.sql.columnar.scanOnly | When enabled, this config will overwrite all other operators' enabling, and only Scan and Filter pushdown will be offloaded to native. [...]
+| spark.gluten.sql.columnar.batchscan | Enable or Disable Columnar BatchScan, default is true [...]
+| spark.gluten.sql.columnar.hashagg | Enable or Disable Columnar Hash Aggregate, default is true [...]
+| spark.gluten.sql.columnar.project | Enable or Disable Columnar Project, default is true [...]
+| spark.gluten.sql.columnar.filter | Enable or Disable Columnar Filter, default is true [...]
+| spark.gluten.sql.columnar.sort | Enable or Disable Columnar Sort, default is true [...]
+| spark.gluten.sql.columnar.window | Enable or Disable Columnar Window, default is true [...]
+| spark.gluten.sql.columnar.shuffledHashJoin | Enable or Disable ShuffledHashJoin, default is true [...]
+| spark.gluten.sql.columnar.forceShuffledHashJoin | Force to use ShuffledHashJoin over SortMergeJoin, default is true. For queries that can benefit from storaged patitioned join, please set it to false. [...]
+| spark.gluten.sql.columnar.sortMergeJoin | Enable or Disable Columnar Sort Merge Join, default is true [...]
+| spark.gluten.sql.columnar.union | Enable or Disable Columnar Union, default is true [...]
+| spark.gluten.sql.columnar.expand | Enable or Disable Columnar Expand, default is true [...]
+| spark.gluten.sql.columnar.generate | Enable or Disable Columnar Generate, default is true [...]
+| spark.gluten.sql.columnar.limit | Enable or Disable Columnar Limit, default is true [...]
+| spark.gluten.sql.columnar.tableCache | Enable or Disable Columnar Table Cache, default is false [...]
+| spark.gluten.sql.columnar.broadcastExchange | Enable or Disable Columnar Broadcast Exchange, default is true [...]
+| spark.gluten.sql.columnar.broadcastJoin | Enable or Disable Columnar BroadcastHashJoin, default is true [...]
+| spark.gluten.sql.columnar.shuffle.sort.threshold | The threshold to determine whether to use sort-based columnar shuffle. Sort-based shuffle will be used if the number of partitions is greater than this threshold. [...]
+| spark.gluten.sql.columnar.shuffle.codec | Set up the codec to be used for Columnar Shuffle. If this configuration is not set, will check the value of spark.io.compression.codec. By default, Gluten use software compression. Valid options for software compression are lz4, zstd. Valid options for QAT and IAA is gzip. [...]
+| spark.gluten.sql.columnar.shuffle.codecBackend | Enable using hardware accelerators for shuffle de/compression. Valid options are QAT and IAA. [...]
+| spark.gluten.sql.columnar.shuffle.compressionMode | Setting different compression mode in shuffle, Valid options are buffer and rowvector, buffer option compress each buffer of RowVector individually into one pre-allocated large buffer, rowvector option first copies each buffer of RowVector to a large buffer and then compress the entire buffer in one go. [...]
+| spark.gluten.sql.columnar.shuffle.compression.threshold | If number of rows in a batch falls below this threshold, will copy all buffers into one buffer to compress. [...]
+| spark.gluten.sql.columnar.shuffle.realloc.threshold | Set the threshold to dynamically adjust the size of shuffle split buffers. The size of each split buffer is recalculated for each incoming batch of data. If the new size deviates from the current partition buffer size by a factor outside the range of [1 - threshold, 1 + threshold], the split buffer will be re-allocated using the newly calculated size [...]
+| spark.gluten.sql.columnar.shuffle.merge.threshold | Set the threshold control the minimum merged size. When a partition buffer is full, and the number of rows is below (`threshold * spark.gluten.sql.columnar.maxBatchSize`), it will be saved for merging. [...]
+| spark.gluten.sql.columnar.numaBinding | Set up NUMABinding, default is false [...]
+| spark.gluten.sql.columnar.coreRange | Set up the core range for NUMABinding, only works when numaBinding set to true. <br /> The setting is based on the number of cores in your system. Use 72 cores as an example. [...]
+| spark.gluten.sql.columnar.wholeStage.fallback.threshold | Configure the threshold for whether whole stage will fall back in AQE supported case by counting the number of ColumnarToRow & vanilla leaf node [...]
+| spark.gluten.sql.columnar.query.fallback.threshold | Configure the threshold for whether query will fall back by counting the number of ColumnarToRow & vanilla leaf node [...]
+| spark.gluten.sql.columnar.fallback.ignoreRowToColumnar | When true, the fallback policy ignores the RowToColumnar when counting fallback number. [...]
+| spark.gluten.sql.columnar.fallback.preferColumnar | When true, the fallback policy prefers to use Gluten plan rather than vanilla Spark plan if the both of them contains ColumnarToRow and the vanilla Spark plan ColumnarToRow number is not smaller than Gluten plan. [...]
+| spark.gluten.sql.columnar.force.hashagg | Force to use hash agg to replace sort agg. [...]
+| spark.gluten.sql.columnar.vanillaReaders | Enable vanilla spark's vectorized reader. Please note it may bring perf. overhead due to extra data transition. We recommend to disable it if most queries can be fully offloaded to gluten. [...]
+| spark.gluten.sql.native.bloomFilter | Enable or Disable native runtime bloom filter. [...]
+| spark.gluten.sql.native.arrow.reader.enabled | Enable or Disable native arrow read CSV file format [...]
+| spark.gluten.shuffleWriter.bufferSize | Set the number of buffer rows for the shuffle writer [...]
+| spark.gluten.loadLibFromJar | Controls whether to load dynamic link library from a packed jar for gluten/cpp. Not applicable to static build and clickhouse backend. [...]
+| spark.gluten.loadLibOS | When `spark.gluten.loadLibFromJar` is true. Manually specify the system os to load library, e.g., CentOS [...]
+| spark.gluten.loadLibOSVersion | Manually specify the system os version to load library, e.g., if `spark.gluten.loadLibOS` is CentOS, this config can be 7 [...]
+| spark.gluten.expression.blacklist | A black list of expression to skip transform, multiple values separated by commas. [...]
+| spark.gluten.sql.columnar.fallback.expressions.threshold | Fall back filter/project if the height of expression tree reaches this threshold, considering Spark codegen can bring better performance for such case. [...]
+| spark.gluten.sql.cartesianProductTransformerEnabled | Config to enable CartesianProductExecTransformer. [...]
+| spark.gluten.sql.broadcastNestedLoopJoinTransformerEnabled | Config to enable BroadcastNestedLoopJoinExecTransformer. [...]
+| spark.gluten.sql.cacheWholeStageTransformerContext | When true, `WholeStageTransformer` will cache the `WholeStageTransformerContext` when executing. It is used to get substrait plan node and native plan string. [...]
+| spark.gluten.sql.injectNativePlanStringToExplain | When true, Gluten will inject native plan tree to explain string inside `WholeStageTransformerContext`. [...]
+| spark.gluten.sql.fallbackRegexpExpressions | When true, Gluten will fall back all regexp expressions to avoid any incompatibility risk. [...]
+
+## Velox Parameters
+
+The following configurations are related to Velox settings.
+
+| Parameters | Description | Recommend Setting |
+|----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
+| spark.gluten.sql.columnar.backend.velox.bloomFilter.expectedNumItems | The default number of expected items for the velox bloomfilter. | 1000000L |
+| spark.gluten.sql.columnar.backend.velox.bloomFilter.numBits | The default number of bits to use for the velox bloom filter. | 8388608L |
+| spark.gluten.sql.columnar.backend.velox.bloomFilter.maxNumBits | The max number of bits to use for the velox bloom filter. | 4194304L |
+| spark.gluten.sql.columnar.backend.velox.fileHandleCacheEnabled | Disables caching if false. File handle cache should be disabled if files are mutable, i.e. file content may change while file path stays the same. | |
+| spark.gluten.sql.columnar.backend.velox.directorySizeGuess | Set the directory size guess for velox file scan. | |
+| spark.gluten.sql.columnar.backend.velox.filePreloadThreshold | Set the file preload threshold for velox file scan. | |
+| spark.gluten.sql.columnar.backend.velox.prefetchRowGroups | Set the prefetch row groups for velox file scan. | |
+| spark.gluten.sql.columnar.backend.velox.loadQuantum | Set the load quantum for velox file scan. | |
+| spark.gluten.sql.columnar.backend.velox.maxCoalescedDistanceBytes | Set the max coalesced distance bytes for velox file scan. | |
+| spark.gluten.sql.columnar.backend.velox.maxCoalescedBytes | Set the max coalesced bytes for velox file scan. | |
+| spark.gluten.sql.columnar.backend.velox.cachePrefetchMinPct | Set prefetch cache min pct for velox file scan. | |
+| spark.gluten.velox.awsSdkLogLevel | Log granularity of AWS C++ SDK in velox. | FATAL |
+| spark.gluten.velox.fs.s3a.retry.mode | Retry mode for AWS s3 connection error, can be "legacy", "standard" and "adaptive". | legacy |
+| spark.gluten.velox.fs.s3a.connect.timeout | Timeout for AWS s3 connection. | 1s |
+| spark.gluten.sql.columnar.backend.velox.orc.scan.enabled | Enable velox orc scan. If disabled, vanilla spark orc scan will be used. | true |
+| spark.gluten.sql.complexType.scan.fallback.enabled | Force fallback for complex type scan, including struct, map, array. | true |
+
+
+Additionally, you can control the configurations of gluten at thread level by local property.
+
+| Parameters | Description | Recommend Setting |
+|--------------------------------|----------------------------------------------|-------------------|
+| gluten.enabledForCurrentThread | Control the usage of gluten at thread level. | true |
+
+Below is an example of developing an application using scala to set local properties.
+
+```scala
+// Before executing the query, set local properties.
+sparkContext.setLocalProperty(key, value)
+spark.sql("select * from demo_tables").show()
+```
diff --git a/docs/Velox.md b/docs/Velox.md
deleted file mode 100644
index a5e9119..0000000
--- a/docs/Velox.md
+++ /dev/null
@@ -1,9 +0,0 @@
----
-layout: page
-title: Velox Backend
-nav_order: 1
-has_children: true
-parent: Documentation
-permalink: /docs/velox/
----
-# Gluten Velox Backend Documents
diff --git a/docs/VeloxBuildGuide.md b/docs/VeloxBuildGuide.md
deleted file mode 100644
index 443778f..0000000
--- a/docs/VeloxBuildGuide.md
+++ /dev/null
@@ -1,78 +0,0 @@
----
-layout: page
-title: Build Parameters for Velox Backend
-nav_order: 2
-parent: Velox Backend
-grand_parent: Documentation
-permalink: /docs/velox/build_parameters
----
-## Build Parameters
-### Native build parameters for buildbundle-veloxbe.sh or builddeps-veloxbe.sh
-Please set them via `--`, e.g. `--build_type=Release`.
-
-| Parameters | Description | Default |
-|------------------------|----------------------------------------------------------------------------|---------|
-| build_type | Build type for Velox & gluten cpp, CMAKE_BUILD_TYPE. | Release |
-| build_tests | Build gluten cpp tests. | OFF |
-| build_examples | Build udf example. | OFF |
-| build_benchmarks | Build gluten cpp benchmarks. | OFF |
-| build_jemalloc | Build with jemalloc. | ON |
-| build_protobuf | Build protobuf lib. | ON |
-| enable_qat | Enable QAT for shuffle data de/compression. | OFF |
-| enable_iaa | Enable IAA for shuffle data de/compression. | OFF |
-| enable_hbm | Enable HBM allocator. | OFF |
-| enable_s3 | Build with S3 support. | OFF |
-| enable_gcs | Build with GCs support. | OFF |
-| enable_hdfs | Build with HDFS support. | OFF |
-| enable_abfs | Build with ABFS support. | OFF |
-| enable_ep_cache | Enable caching for external project build (Velox). | OFF |
-| enable_vcpkg | Enable vcpkg for static build. | OFF |
-| run_setup_script | Run setup script to install Velox dependencies. | ON |
-| velox_repo | Specify your own Velox repo to build. | "" |
-| velox_branch | Specify your own Velox branch to build. | "" |
-| velox_home | Specify your own Velox source path to build. | "" |
-| build_velox_tests | Build Velox tests. | OFF |
-| build_velox_benchmarks | Build Velox benchmarks (velox_tests and connectors will be disabled if ON) | OFF |
-| compile_arrow_java | Compile arrow java for gluten build to use to fix invalid pointer issues. | OFF |
-
-### Velox build parameters for build_velox.sh
-Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
-
-| Parameters | Description | Default |
-|--------------------|-------------------------------------------------------------------------|------------------------------------------|
-| velox_home | Specify Velox source path to build. | GLUTEN_SRC/ep/build-velox/build/velox_ep |
-| build_type | Velox build type, i.e., CMAKE_BUILD_TYPE. | Release |
-| enable_s3 | Build Velox with S3 support. | OFF |
-| enable_gcs | Build Velox with GCS support. | OFF |
-| enable_hdfs | Build Velox with HDFS support. | OFF |
-| enable_abfs | Build Velox with ABFS support. | OFF |
-| run_setup_script | Run setup script to install Velox dependencies before build. | ON |
-| enable_ep_cache | Enable and reuse cache of Velox build. | OFF |
-| build_test_utils | Build Velox with cmake arg -DVELOX_BUILD_TEST_UTILS=ON if ON. | OFF |
-| build_tests | Build Velox test. | OFF |
-| build_benchmarks | Build Velox benchmarks. | OFF |
-| compile_arrow_java | Build arrow java for gluten build to use to fix invalid pointer issues. | OFF |
-
-### Maven build parameters
-The below parameters can be set via `-P` for mvn.
-
-| Parameters | Description | Default state |
-|---------------------|------------------------------------------------------------------------------|---------------|
-| backends-velox | Build Gluten Velox backend. | disabled |
-| backends-clickhouse | Build Gluten ClickHouse backend. | disabled |
-| rss | Build Gluten with Remote Shuffle Service, only applicable for Velox backend. | disabled |
-| delta | Build Gluten with Delta Lake support. | disabled |
-| iceberg | Build Gluten with Iceberg support. | disabled |
-| spark-3.2 | Build Gluten for Spark 3.2. | enabled |
-| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
-| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
-
-## Gluten Jar for Deployment
-The gluten jar built out is under `GLUTEN_SRC/package/target/`.
-It's name pattern is `gluten-<backend_type>-bundle-spark<spark.bundle.version>_<scala.binary.version>-<os.detected.release>_<os.detected.release.version>-<project.version>.jar`.
-
-| Spark Version | spark.bundle.version | scala.binary.version |
-|---------------|----------------------|----------------------|
-| 3.2.2 | 3.2 | 2.12 |
-| 3.3.1 | 3.3 | 2.12 |
-| 3.4.2 | 3.4 | 2.12 |
diff --git a/developers/CppCodingStyle.md b/docs/developers/CppCodingStyle.md
similarity index 99%
rename from developers/CppCodingStyle.md
rename to docs/developers/CppCodingStyle.md
index a75feba..ba20092 100644
--- a/developers/CppCodingStyle.md
+++ b/docs/developers/CppCodingStyle.md
@@ -3,6 +3,8 @@ layout: page
title: CPP Code Style
nav_order: 4
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/cpp-code-style/
---
# Gluten CPP Core Guidelines
diff --git a/developers/HowTo.md b/docs/developers/HowTo.md
similarity index 99%
rename from developers/HowTo.md
rename to docs/developers/HowTo.md
index 81d48db..9c7401a 100644
--- a/developers/HowTo.md
+++ b/docs/developers/HowTo.md
@@ -1,8 +1,10 @@
---
layout: page
title: How To Use Gluten
-nav_order: 2
+nav_order: 1
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/how-to-use/
---
There are some common questions about developing, debugging and testing been asked again and again. In order to help the developers to contribute
to Gluten as soon as possible, we collected these frequently asked questions, and organized them in the form of Q&A. It's convenient for the developers
diff --git a/developers/HowToRelease.md b/docs/developers/HowToRelease.md
similarity index 100%
rename from developers/HowToRelease.md
rename to docs/developers/HowToRelease.md
diff --git a/developers/MicroBenchmarks.md b/docs/developers/MicroBenchmarks.md
similarity index 99%
rename from developers/MicroBenchmarks.md
rename to docs/developers/MicroBenchmarks.md
index 25fb86b..af99705 100644
--- a/developers/MicroBenchmarks.md
+++ b/docs/developers/MicroBenchmarks.md
@@ -3,6 +3,8 @@ layout: page
title: Micro Benchmarks for Velox Backend
nav_order: 3
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/microbenchmarks/
---
# Generate Micro Benchmarks for Velox Backend
diff --git a/developers/NewToGluten.md b/docs/developers/NewToGluten.md
similarity index 99%
rename from developers/NewToGluten.md
rename to docs/developers/NewToGluten.md
index e12ff0b..bdd83b6 100644
--- a/developers/NewToGluten.md
+++ b/docs/developers/NewToGluten.md
@@ -1,8 +1,10 @@
---
layout: page
title: New To Gluten
-nav_order: 1
+nav_order: 2
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/new-to/
---
Help users to debug and test with gluten.
diff --git a/developers/SubstraitModifications.md b/docs/developers/SubstraitModifications.md
similarity index 96%
rename from developers/SubstraitModifications.md
rename to docs/developers/SubstraitModifications.md
index 0f850fe..75e0c55 100644
--- a/developers/SubstraitModifications.md
+++ b/docs/developers/SubstraitModifications.md
@@ -3,6 +3,8 @@ layout: page
title: Substrait Modifications
nav_order: 9
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/substrait/
---
# Substrait Modifications in Gluten
diff --git a/developers/UsingGperftoolsInCH.md b/docs/developers/UsingGperftoolsInCH.md
similarity index 100%
rename from developers/UsingGperftoolsInCH.md
rename to docs/developers/UsingGperftoolsInCH.md
diff --git a/developers/UsingJemallocWithCH.md b/docs/developers/UsingJemallocWithCH.md
similarity index 100%
rename from developers/UsingJemallocWithCH.md
rename to docs/developers/UsingJemallocWithCH.md
diff --git a/developers/docker_centos7.md b/docs/developers/docker_centos7.md
similarity index 93%
rename from developers/docker_centos7.md
rename to docs/developers/docker_centos7.md
index 80461ca..8c11e00 100644
--- a/developers/docker_centos7.md
+++ b/docs/developers/docker_centos7.md
@@ -1,8 +1,10 @@
---
layout: page
title: Docker script for CentOS 7
-nav_order: 8
+nav_order: 7
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/docker-centos7/
---
Here is a docker script we verified to build Gluten+Velox backend on CentOS 7:
diff --git a/developers/docker_centos8.md b/docs/developers/docker_centos8.md
similarity index 94%
rename from developers/docker_centos8.md
rename to docs/developers/docker_centos8.md
index 4810abd..20a2ff2 100755
--- a/developers/docker_centos8.md
+++ b/docs/developers/docker_centos8.md
@@ -1,8 +1,10 @@
---
layout: page
title: Docker script for CentOS 8
-nav_order: 7
+nav_order: 8
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/docker-centos8/
---
Here is a docker script we verified to build Gluten+Velox backend on Centos8:
diff --git a/developers/docker_ubuntu22.04.md b/docs/developers/docker_ubuntu22.04.md
similarity index 96%
rename from developers/docker_ubuntu22.04.md
rename to docs/developers/docker_ubuntu22.04.md
index 2330dca..64364af 100644
--- a/developers/docker_ubuntu22.04.md
+++ b/docs/developers/docker_ubuntu22.04.md
@@ -3,6 +3,8 @@ layout: page
title: Docker script for Ubuntu 22.04/20.04
nav_order: 6
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/docker-ubuntu/
---
To the first build, it's suggested to build Gluten in a clean docker image. Otherwise it's easy to run into library version conflict issues.
diff --git a/developers/index.md b/docs/developers/index.md
similarity index 77%
rename from developers/index.md
rename to docs/developers/index.md
index e7728aa..14a367f 100644
--- a/developers/index.md
+++ b/docs/developers/index.md
@@ -3,7 +3,8 @@ layout: page
title: Developers
nav_order: 3
has_children: true
-permalink: /developers/
+parent: Documentations
+permalink: /docs/developers/
---
# Gluten Developers
This document provides a developer overview of the project and covers the
diff --git a/developers/velox-function-development-guide.md b/docs/developers/velox-function-development-guide.md
similarity index 97%
rename from developers/velox-function-development-guide.md
rename to docs/developers/velox-function-development-guide.md
index 71d80a7..84d1713 100644
--- a/developers/velox-function-development-guide.md
+++ b/docs/developers/velox-function-development-guide.md
@@ -3,6 +3,8 @@ layout: page
title: Velox Function Development
nav_order: 5
parent: Developers
+grand_parent: Documentations
+permalink: /docs/developers/velox-function-dev/
---
# Developer Guide for Implementing Spark Built-in SQL Functions in Velox
diff --git a/docs/GettingStarted_ClickHouse.md b/docs/get-started/ClickHouse.md
similarity index 91%
rename from docs/GettingStarted_ClickHouse.md
rename to docs/get-started/ClickHouse.md
index b03ba4a..9d880b4 100644
--- a/docs/GettingStarted_ClickHouse.md
+++ b/docs/get-started/ClickHouse.md
@@ -1,10 +1,10 @@
---
layout: page
-title: Getting Started with ClickHouse Backend
-nav_order: 1
-parent: ClickHouse Backend
-grand_parent: Documentation
-permalink: /docs/clickhouse/getting-started
+title: Getting start with ClickHouse Backend
+nav_order: 2
+parent: Getting-Started
+grand_parent: Documentations
+permalink: /docs/getting-started/clickhouse-backend/
---
## ClickHouse Backend
@@ -45,7 +45,7 @@ You need to install the following software manually:
Then, get Gluten code:
```
- git clone https://github.com/oap-project/gluten.git
+ git clone https://github.com/apache/incubator-gluten.git
```
#### Setup ClickHouse backend development environment
@@ -107,8 +107,8 @@ Otherwise, do:
In case you don't want a develop environment, you can use the following command to compile ClickHouse backend directly:
```
-git clone https://github.com/oap-project/gluten.git
-cd gluten
+git clone https://github.com/apache/incubator-gluten.git
+cd incubator-gluten
bash ./ep/build-clickhouse/src/build_clickhouse.sh
```
@@ -124,8 +124,8 @@ The prerequisites are the same as the one mentioned above. Compile Gluten with C
- for Spark 3.2.2<span id="deploy-spark-322"></span>
```
- git clone https://github.com/oap-project/gluten.git
- cd gluten/
+ git clone https://github.com/apache/incubator-gluten.git
+ cd incubator-gluten/
export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
mvn clean install -Pbackends-clickhouse -Phadoop-2.7.4 -Pspark-3.2 -Dhadoop.version=2.8.5 -DskipTests -Dcheckstyle.skip
ls -al backends-clickhouse/target/gluten-XXXXX-spark-3.2-jar-with-dependencies.jar
@@ -134,8 +134,8 @@ The prerequisites are the same as the one mentioned above. Compile Gluten with C
- for Spark 3.3.1
```
- git clone https://github.com/oap-project/gluten.git
- cd gluten/
+ git clone https://github.com/apache/incubator-gluten.git
+ cd incubator-gluten/
export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
mvn clean install -Pbackends-clickhouse -Phadoop-2.7.4 -Pspark-3.3 -Dhadoop.version=2.8.5 -DskipTests -Dcheckstyle.skip
ls -al backends-clickhouse/target/gluten-XXXXX-spark-3.3-jar-with-dependencies.jar
@@ -192,7 +192,7 @@ cd spark-3.2.2-bin-hadoop2.7
--conf spark.sql.columnVector.offheap.enabled=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=6442450944 \
- --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.sql.columnar.columnarToRow=true \
--conf spark.executorEnv.LD_PRELOAD=/path_to_clickhouse_library/libch.so\
--conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
@@ -373,7 +373,7 @@ cd spark-3.2.2-bin-hadoop2.7
--conf spark.sql.columnVector.offheap.enabled=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=6442450944 \
- --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.sql.columnar.columnarToRow=true \
--conf spark.executorEnv.LD_PRELOAD=/path_to_clickhouse_library/libch.so\
--conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
@@ -456,7 +456,7 @@ $spark_cmd \
--conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
--conf spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf=$hdfs_conf \
--conf spark.gluten.sql.columnar.backend.ch.runtime_config.logger.level=debug \
- --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.executorEnv.LD_PRELOAD=$LD_PRELOAD \
--conf spark.hadoop.input.connect.timeout=600000 \
--conf spark.hadoop.input.read.timeout=600000 \
@@ -565,7 +565,7 @@ cd spark-3.2.2-bin-hadoop2.7
--conf spark.memory.offHeap.size=42949672960 \
--conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
--conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.sql.columnar.columnarToRow=true \
--conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
--conf spark.gluten.sql.columnar.iterator=true \
@@ -631,19 +631,26 @@ public read-only account：gluten/hN2xX3uQ4m
### Celeborn support
-Gluten with clickhouse backend has not yet supportted [Celeborn](https://github.com/apache/incubator-celeborn) natively as remote shuffle service using columar shuffle. However, you can still use Celeborn with row shuffle, which means a ColumarBatch will be converted to a row during shuffle.
-Below introduction is used to enable this feature:
+Gluten with clickhouse backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x`, `0.4.x` and `0.5.0`.
-First refer to this URL(https://github.com/apache/incubator-celeborn) to setup a celeborn cluster.
+Below introduction is used to enable this feature.
+
+First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster.
+
+When compiling the Gluten Java module, it's required to enable `celeborn` profile, as follows:
+
+```
+mvn clean package -Pbackends-clickhouse -Pspark-3.3 -Pceleborn -DskipTests
+```
Then add the Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
-- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
+- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
-Currently to use Celeborn following configurations are required in `spark-defaults.conf`
+Currently to use Gluten following configurations are required in `spark-defaults.conf`
```
-spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
+spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
# celeborn master
spark.celeborn.master.endpoints clb-master:9097
@@ -668,27 +675,10 @@ spark.sql.adaptive.localShuffleReader.enabled false
spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
# If you want to use dynamic resource allocation,
-# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+# please refer to this URL (https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
spark.dynamicAllocation.enabled false
```
-#### Celeborn Columnar Shuffle Support
-The native Celeborn support can be enabled by the following configuration
-```
-spark.shuffle.manager=org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
-```
-
-quickly start a celeborn cluster
-```shell
-wget https://dlcdn.apache.org/incubator/celeborn/celeborn-0.3.0-incubating/apache-celeborn-0.3.0-incubating-bin.tgz && \
-tar -zxvf apache-celeborn-0.3.0-incubating-bin.tgz && \
-mv apache-celeborn-0.3.0-incubating-bin/conf/celeborn-defaults.conf.template apache-celeborn-0.3.0-incubating-bin/conf/celeborn-defaults.conf && \
-mv apache-celeborn-0.3.0-incubating-bin/conf/log4j2.xml.template apache-celeborn-0.3.0-incubating-bin/conf/log4j2.xml && \
-mkdir /opt/hadoop && chmod 777 /opt/hadoop && \
-echo -e "celeborn.worker.flusher.threads 4\nceleborn.worker.storage.dirs /tmp\nceleborn.worker.monitor.disk.enabled false" > apache-celeborn-0.3.0-incubating-bin/conf/celeborn-defaults.conf && \
-bash apache-celeborn-0.3.0-incubating-bin/sbin/start-master.sh && bash apache-celeborn-0.3.0-incubating-bin/sbin/start-worker.sh
-```
-
### Columnar shuffle mode
We have two modes of columnar shuffle
1. prefer cache
diff --git a/docs/GettingStarted_Velox.md b/docs/get-started/Velox.md
similarity index 73%
rename from docs/GettingStarted_Velox.md
rename to docs/get-started/Velox.md
index 086c429..7b88608 100644
--- a/docs/GettingStarted_Velox.md
+++ b/docs/get-started/Velox.md
@@ -1,34 +1,29 @@
---
layout: page
-title: Getting-Started with Velox Backend
+title: Getting Start with Velox Backend
nav_order: 1
-parent: Velox Backend
-grand_parent: Documentation
-permalink: /docs/velox/getting-started
+parent: Getting-Started
+grand_parent: Documentations
+permalink: /docs/getting-started/velox-backend/
---
-# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8 |
-| scala | 2.12
-
-Spark3.4.0 support is still WIP. TPCH/DS can pass, UT is not yet passed.
-
-There are pending PRs for jdk11 support.
+# Supported Version
-Currently, the mvn script can automatically fetch and build all dependency libraries incluing Velox. Our nightly build still use Velox under oap-project.
+| Type | Version |
+|-------|---------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.2, 3.5.1 |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
# Prerequisite
-Currently, Gluten+Velox backend is only tested on **Ubuntu20.04/Ubuntu22.04/Centos8**. Other kinds of OS support are still in progress. The long term goal is to support several
-common OS and conda env deployment.
+Currently, Gluten+Velox backend is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**.
+Other kinds of OS support are still in progress. The long term goal is to support several common OS and conda env deployment.
-Gluten builds with Spark3.2.x and Spark3.3.x now but only fully tested in CI with 3.2.2 and 3.3.1. We will add/update supported/tested versions according to the upstream changes.
+Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.2 and 3.5.1.
-we need to set up the `JAVA_HOME` env. Currently, **java 8** is required and the support for java 11/17 is not ready.
+We need to set up the `JAVA_HOME` env. Currently, Gluten supports **java 8** and **java 17**.
**For x86_64**
@@ -52,7 +47,7 @@ export PATH=$JAVA_HOME/bin:$PATH
## config maven, like proxy in ~/.m2/settings.xml
## fetch gluten code
-git clone https://github.com/oap-project/gluten.git
+git clone https://github.com/apache/incubator-gluten.git
```
# Build Gluten with Velox Backend
@@ -62,16 +57,17 @@ It's recommended to use buildbundle-veloxbe.sh to build gluten in one script.
**For x86_64 build**
-```bash
-cd /path/to/gluten
+First time build for all supported spark versions.
-## The script builds two jars for spark 3.2.2 and 3.3.1.
+```bash
./dev/buildbundle-veloxbe.sh
+```
+
+After a complete build, if only some gluten code is changed, you can use the following command to skip building velox/arrow and
+setting up build dependencies.
-## After a complete build, if you need to re-build the project and only some gluten code is changed,
-## you can use the following command to skip building velox and protobuf.
-# ./dev/buildbundle-veloxbe.sh --enable_ep_cache=ON --build_protobuf=OFF
-## If you have the same error with issue-3283, you need to add the parameter `--compile_arrow_java=ON`
+```bash
+./dev/buildbundle-veloxbe.sh --enable_ep_cache=ON --build_arrow=OFF --run_setup_script=OFF
```
**For aarch64 build:**
@@ -79,53 +75,52 @@ cd /path/to/gluten
```bash
export CPU_TARGET="aarch64"
-cd /path/to/gluten
-
./dev/builddeps-veloxbe.sh
```
**Build Velox separately**
+Currently, Gluten is using a [forked Velox](https://github.com/oap-project/velox/) which is daily updated based on [upstream Velox](https://github.com/facebookincubator/velox).
+
Scripts under `/path/to/gluten/ep/build-velox/src` provide `get_velox.sh` and `build_velox.sh` to build Velox separately, you could use these scripts with custom repo/branch/location.
Velox provides arrow/parquet lib. Gluten cpp module need a required VELOX_HOME parsed by --velox_home, if you specify custom ep location, make sure these variables be passed correctly.
```bash
## fetch Velox and compile
-cd /path/to/gluten/ep/build-velox/src/
-## you could use custom ep location by --velox_home=custom_path, make sure specify --velox_home in build_velox.sh too.
-./get_velox.sh
-## make sure specify --velox_home if you have specified it in get_velox.sh.
-./build_velox.sh
+./dev/builddeps-veloxbe.sh build_velox
## compile Gluten cpp module
-cd /path/to/gluten/cpp
-## if you use custom velox_home, make sure specified here by --velox_home
-./compile.sh --build_velox_backend=ON
+./dev/builddeps-veloxbe.sh build_gluten_cpp
## compile Gluten java module and create package jar
cd /path/to/gluten
# For spark3.2.x
-mvn clean package -Pbackends-velox -Prss -Pspark-3.2 -DskipTests
+mvn clean package -Pbackends-velox -Pceleborn -Puniffle -Pspark-3.2 -DskipTests
# For spark3.3.x
-mvn clean package -Pbackends-velox -Prss -Pspark-3.3 -DskipTests
+mvn clean package -Pbackends-velox -Pceleborn -Puniffle -Pspark-3.3 -DskipTests
+# For spark3.4.x
+mvn clean package -Pbackends-velox -Pceleborn -Puniffle -Pspark-3.4 -DskipTests
+# For spark3.5.x
+mvn clean package -Pbackends-velox -Pceleborn -Puniffle -Pspark-3.5 -DskipTests
```
-notes：The compilation of `Velox` using the script of `build_velox.sh` may fail caused by `oom`, you can prevent this failure by using the user command of `export NUM_THREADS=4` before executing the above scripts.
+Notes： Building Velox may fail caused by `oom`. You can prevent this failure by adjusting `NUM_THREADS` (e.g., `export NUM_THREADS=4`) before building Gluten/Velox.
-Once building successfully, the Jar file will be generated in the directory: package/target/\<gluten-jar\> for Spark 3.2.x/Spark 3.3.x.
+Once building successfully, the Jar file will be generated in the directory: package/target/\<gluten-jar\> for Spark 3.2.x/Spark 3.3.x/Spark 3.4.x/Spark 3.5.x.
## Dependency library deployment
With config `enable_vcpkg=ON`, the dependency libraries will be built and statically linked into libvelox.so and libgluten.so, which is packed into the gluten-jar. In this way, only the gluten-jar is needed to add to `spark.<driver|executor>.extraClassPath` and spark will deploy the jar to each worker node. It's better to build the static version using a clean docker image without any extra libraries installed. On host with some libraries like jemalloc installed, the script may crash wit [...]
-With config `enable_vcpkg=OFF`, the dependency libraries won't be statically linked, instead the script will install the libraries to system then pack the dependency libraries into another jar named gluten-package-${Maven-artifact-version}.jar. Then you need to add the jar to extraClassPath then set `spark.gluten.loadLibFromJar=true`. Or you already manually deployed the dependency libraries on each worker node. You may find the libraries list from the gluten-package jar.
+With config `enable_vcpkg=OFF`, not all dependency libraries will be statically linked, instead the script will install the libraries to system then pack the dependency libraries into another jar named `gluten-package-${Maven-artifact-version}.jar`. Then you need to add the jar to `extraClassPath` and set `spark.gluten.loadLibFromJar=true`. Otherwise, you need to install shared dependency libraries on each worker node. You may find the libraries list from the gluten-package jar.
## HDFS support
Hadoop hdfs support is ready via the [libhdfs3](https://github.com/apache/hawq/tree/master/depends/libhdfs3) library. The libhdfs3 provides native API for Hadoop I/O without the drawbacks of JNI. It also provides advanced authentication like Kerberos based. Please note this library has several dependencies which may require extra installations on Driver and Worker node.
### Build with HDFS support
+
To build Gluten with HDFS support, below command is suggested:
```bash
@@ -134,7 +129,7 @@ cd /path/to/gluten
```
### Configuration about HDFS support
-
+
HDFS uris (hdfs://host:port) will be extracted from a valid hdfs file path to initialize hdfs client, you do not need to specify it explicitly.
libhdfs3 need a configuration file and [example here](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/src/backend/utils/misc/etc/hdfs-client.xml), this file is a bit different from hdfs-site.xml and core-site.xml.
@@ -154,7 +149,7 @@ cp /path/to/hdfs-client.xml hdfs-client.xml
One typical deployment on Spark/HDFS cluster is to enable [short-circuit reading](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html). Short-circuit reads provide a substantial performance boost to many applications.
-By default libhdfs3 does not set the default hdfs domain socket path to support HDFS short-circuit read. If this feature is required in HDFS setup, users may need to setup the domain socket path correctly by patching the libhdfs3 source code or by setting the correct config environment. In Gluten the short-circuit domain socket path is set to "/var/lib/hadoop-hdfs/dn_socket" in [build_velox.sh](https://github.com/oap-project/gluten/blob/main/ep/build-velox/src/build_velox.sh) So we need [...]
+By default libhdfs3 does not set the default hdfs domain socket path to support HDFS short-circuit read. If this feature is required in HDFS setup, users may need to setup the domain socket path correctly by patching the libhdfs3 source code or by setting the correct config environment. In Gluten the short-circuit domain socket path is set to "/var/lib/hadoop-hdfs/dn_socket" in [build_velox.sh](https://github.com/apache/incubator-gluten/blob/main/ep/build-velox/src/build_velox.sh) So we [...]
```
sudo mkdir -p /var/lib/hadoop-hdfs/
@@ -165,12 +160,12 @@ You also need to add configuration to the "hdfs-site.xml" as below:
```
<property>
- <name>dfs.client.read.shortcircuit</name>
- <value>true</value>
+ <name>dfs.client.read.shortcircuit</name>
+ <value>true</value>
</property>
<property>
- <name>dfs.domain.socket.path</name>
- <value>/var/lib/hadoop-hdfs/dn_socket</value>
+ <name>dfs.domain.socket.path</name>
+ <value>/var/lib/hadoop-hdfs/dn_socket</value>
</property>
```
@@ -180,10 +175,10 @@ Here are two steps to enable kerberos.
- Make sure the hdfs-client.xml contains
-```xml
+```
<property>
- <name>hadoop.security.authentication</name>
- <value>kerberos</value>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
</property>
```
@@ -195,6 +190,19 @@ Here are two steps to enable kerberos.
The ticket cache file can be found by `klist`.
+## Azure Blob File System (ABFS) support
+
+Velox supports ABFS with the open source [Azure SDK for C++](https://github.com/Azure/azure-sdk-for-cpp) and Gluten uses the Velox ABFS connector to connect with ABFS.
+The build option for ABFS (enable_abfs) must be set to enable this feature as listed below.
+
+```
+cd /path/to/gluten
+./dev/buildbundle-veloxbe.sh --enable_abfs=ON
+```
+
+Please refer [Velox ABFS](VeloxABFS.md) part for more detailed configurations.
+
+
## AWS S3 support
Velox supports S3 with the open source [AWS C++ SDK](https://github.com/aws/aws-sdk-cpp) and Gluten uses Velox S3 connector to connect with S3.
@@ -209,20 +217,22 @@ Currently there are several ways to asscess S3 in Spark. Please refer [Velox S3]
## Celeborn support
-Gluten with velox backend supports [Celeborn](https://github.com/apache/incubator-celeborn) as remote shuffle service. Below introduction is used to enable this feature
+Gluten with velox backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x`, `0.4.x` and `0.5.0`.
+
+Below introduction is used to enable this feature.
-First refer to this URL(https://github.com/apache/incubator-celeborn) to setup a celeborn cluster.
+First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster.
-When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+When compiling the Gluten Java module, it's required to enable `celeborn` profile, as follows:
```
-mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+mvn clean package -Pbackends-velox -Pspark-3.3 -Pceleborn -DskipTests
```
Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
-- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
-- Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-thirdparty-lib-xx-xx.jar
+- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
+- Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-celeborn-package-xx-SNAPSHOT.jar
Currently to use Gluten following configurations are required in `spark-defaults.conf`
@@ -252,7 +262,41 @@ spark.sql.adaptive.localShuffleReader.enabled false
spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
# If you want to use dynamic resource allocation,
-# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+# please refer to this URL (https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+spark.dynamicAllocation.enabled false
+```
+
+## Uniffle support
+
+Uniffle with velox backend supports [Uniffle](https://github.com/apache/incubator-uniffle) as remote shuffle service. Currently, the supported Uniffle versions are `0.8.0`.
+
+First refer to this URL(https://uniffle.apache.org/docs/intro) to get start with uniffle.
+
+When compiling the Gluten Java module, it's required to enable `uniffle` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Puniffle -DskipTests
+```
+
+Then add the Uniffle and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Uniffle: rss-client-spark3-shaded-[uniffleVersion].jar
+- Gluten: gluten-uniffle-velox-xxx-SNAPSHOT-3.x.jar
+
+Currently to use Gluten following configurations are required in `spark-defaults.conf`
+
+```
+spark.shuffle.manager org.apache.spark.shuffle.gluten.uniffle.UniffleShuffleManager
+# uniffle coordinator address
+spark.rss.coordinator.quorum ip:port
+# Support for Spark AQE
+spark.sql.adaptive.localShuffleReader.enabled false
+spark.shuffle.service.enabled false
+# Uniffle support mutilple storage types, you can choose one of them.
+# Such as MEMORY,LOCALFILE,MEMORY_LOCALFILE,HDFS,MEMORY_HDFS,LOCALFILE_HDFS,MEMORY_LOCALFILE_HDFS
+spark.rss.storage.type LOCALFILE_HDFS
+# If you want to use dynamic resource allocation,
+# please refer to this URL (https://github.com/apache/incubator-uniffle/tree/master/patch/spark) to apply the patch into your own Spark.
spark.dynamicAllocation.enabled false
```
@@ -268,7 +312,7 @@ First of all, compile gluten-delta module by a `delta` profile, as follows:
mvn clean package -Pbackends-velox -Pspark-3.3 -Pdelta -DskipTests
```
-Then, put the additional gluten-delta jar to the class path (usually it's `$SPARK_HOME/jars`).
+Then, put the additional `gluten-delta-XX-SNAPSHOT.jar` to the class path (usually it's `$SPARK_HOME/jars`).
The gluten-delta jar is in `gluten-delta/target` directory.
After the two steps, you can query delta table by gluten/velox without scan's fallback.
@@ -288,20 +332,20 @@ First of all, compile gluten-iceberg module by a `iceberg` profile, as follows:
mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests
```
-Then, put the additional gluten-iceberg jar to the class path (usually it's `$SPARK_HOME/jars`).
-The gluten-iceberg jar is in `gluten-iceberg/target` directory.
+Once built successfully, iceberg features will be included in gluten-velox-bundle-X jar. Then you can query iceberg table by gluten/velox without scan's fallback.
After the two steps, you can query iceberg table by gluten/velox without scan's fallback.
# Coverage
-Spark3.3 has 387 functions in total. ~240 are commonly used. Velox's functions have two category, Presto and Spark. Presto has 124 functions implemented. Spark has 62 functions. Spark functions are verified to have the same result as Vanilla Spark. Some Presto functions have the same result as Vanilla Spark but some others have different. Gluten prefer to use Spark functions firstly. If it's not in Spark's list but implemented in Presto, we currently offload to Presto one until we noted [...]
+
+Spark3.3 has 387 functions in total. ~240 are commonly used. To get the support status of all Spark built-in functions, please refer to [Velox Backend's Supported Operators & Functions](../velox-backend-support-progress.md).
> Velox doesn't support [ANSI mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)), so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will fallback to Vanilla Spark.
To identify what can be offloaded in a query and detailed fallback reasons, user can follow below steps to retrieve corresponding logs.
```
-1) Enable Gluten by proper [configuration](https://github.com/oap-project/gluten/blob/main/docs/Configuration.md).
+1) Enable Gluten by proper [configuration](https://github.com/apache/incubator-gluten/blob/main/docs/Configuration.md).
2) Disable Spark AQE to trigger plan validation in Gluten
spark.sql.adaptive.enabled = false
@@ -345,142 +389,16 @@ Using the following configuration options to customize spilling:
| spark.gluten.sql.columnar.backend.velox.aggregationSpillEnabled | true | Whether spill is enabled on aggregations |
| spark.gluten.sql.columnar.backend.velox.joinSpillEnabled | true | Whether spill is enabled on joins |
| spark.gluten.sql.columnar.backend.velox.orderBySpillEnabled | true | Whether spill is enabled on sorts |
-| spark.gluten.sql.columnar.backend.velox.aggregationSpillMemoryThreshold | 0 | Memory limit before spilling to disk for aggregations, per Spark task. Unit: byte |
-| spark.gluten.sql.columnar.backend.velox.joinSpillMemoryThreshold | 0 | Memory limit before spilling to disk for joins, per Spark task. Unit: byte |
-| spark.gluten.sql.columnar.backend.velox.orderBySpillMemoryThreshold | 0 | Memory limit before spilling to disk for sorts, per Spark task. Unit: byte |
| spark.gluten.sql.columnar.backend.velox.maxSpillLevel | 4 | The max allowed spilling level with zero being the initial spilling level |
-| spark.gluten.sql.columnar.backend.velox.maxSpillFileSize | 20MB | The max allowed spill file size. If it is zero, then there is no limit |
-| spark.gluten.sql.columnar.backend.velox.minSpillRunSize | 268435456 | The min spill run size limit used to select partitions for spilling |
+| spark.gluten.sql.columnar.backend.velox.maxSpillFileSize | 1GB | The max allowed spill file size. If it is zero, then there is no limit |
| spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit | 29 | The start partition bit which is used with 'spillPartitionBits' together to calculate the spilling partition number |
| spark.gluten.sql.columnar.backend.velox.spillPartitionBits | 2 | The number of bits used to calculate the spilling partition number. The number of spilling partitions will be power of two |
| spark.gluten.sql.columnar.backend.velox.spillableReservationGrowthPct | 25 | The spillable memory reservation growth percentage of the previous memory reservation size |
+| spark.gluten.sql.columnar.backend.velox.spillThreadNum | 0 | (Experimental) The thread num of a dedicated thread pool to do spill
-# Velox User-Defined Functions (UDF)
-
-## Introduction
-
-Velox backend supports User-Defined Functions (UDF). Users can create their own functions using the UDF interface provided in Velox backend and build libraries for these functions. At runtime, the UDF are registered at the start of applications. Once registered, Gluten will be able to parse and offload these UDF into Velox during execution.
-
-## Creating a UDF library
-
-The following steps demonstrate how to set up a UDF library project:
-
-- **Include the UDF Interface Header:** First, include the UDF interface header file [Udf.h](../../cpp/velox/udf/Udf.h) in the project file. The header file defines the `UdfEntry` struct, along with the macros for declaring the necessary functions to integrate the UDF into Gluten and Velox.
+# Velox User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAF)
-- **Implement the UDF:** Implement UDF. These functions should be able to register to Velox.
-
-- **Implement the Interface Functions:** Implement the following interface functions that integrate UDF into Project Gluten:
-
- - `getNumUdf()`: This function should return the number of UDF in the library. This is used to allocating udfEntries array as the argument for the next function `getUdfEntries`.
-
- - `getUdfEntries(gluten::UdfEntry* udfEntries)`: This function should populate the provided udfEntries array with the details of the UDF, including function names and return types.
-
- - `registerUdf()`: This function is called to register the UDF to Velox function registry. This is where users should register functions by calling `facebook::velox::exec::registerVecotorFunction` or other Velox APIs.
-
- - The interface functions are mapping to marcos in [Udf.h](../../cpp/velox/udf/Udf.h). Here's an example of how to implement these functions:
-
- ```
- // Filename MyUDF.cpp
-
- #include <velox/expression/VectorFunction.h>
- #include <velox/udf/Udf.h>
-
- const int kNumMyUdf = 1;
-
- gluten::UdfEntry myUdf[kNumMyUdf] = {{"myudf1", "integer"}};
-
- class MyUdf : public facebook::velox::exec::VectorFunction {
- ... // Omit concrete implementation
- }
-
- static std::vector<std::shared_ptr<exec::FunctionSignature>>
- myUdfSignatures() {
- return {facebook::velox::exec::FunctionSignatureBuilder()
- .returnType(myUdf[0].dataType)
- .argumentType("integer")
- .build()};
- }
-
- DEFINE_GET_NUM_UDF { return kNumMyUdf; }
-
- DEFINE_GET_UDF_ENTRIES { udfEntries[0] = myUdf[0]; }
-
- DEFINE_REGISTER_UDF {
- facebook::velox::exec::registerVectorFunction(
- myUdf[0].name, myUdfSignatures(), std::make_unique<MyUdf>());
- }
-
- ```
-
-## Building the UDF library
-
-To build the UDF library, users need to compile the C++ code and link to `libvelox.so`. It's recommended to create a CMakeLists.txt for the project. Here's an example:
-```
-project(myudf)
-
-set(CMAKE_CXX_STANDARD 17)
-set(CMAKE_CXX_STANDARD_REQUIRED ON)
-
-set(GLUTEN_HOME /path/to/gluten)
-
-add_library(myudf SHARED "MyUDF.cpp")
-
-find_library(VELOX_LIBRARY REQUIRED NAMES velox HINTS ${GLUTEN_HOME}/cpp/build/releases NO_DEFAULT_PATH)
-
-target_include_directories(myudf PRIVATE ${GLUTEN_HOME}/cpp ${GLUTEN_HOME}/ep/build-velox/build/velox_ep)
-target_link_libraries(myudf PRIVATE ${VELOX_LIBRARY})
-```
-
-## Using UDF in Gluten
-
-Gluten loads the UDF libraries at runtime. You can upload UDF libraries via `--files` or `--archives`, and configure the libray paths using the provided Spark configuration, which accepts comma separated list of library paths.
-
-Note if running on Yarn client mode, the uploaded files are not reachable on driver side. Users should copy those files to somewhere reachable for driver and set `spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths`. This configuration is also useful when the `udfLibraryPaths` is different between driver side and executor side.
-
-- Use `--files`
-```shell
---files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
---conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
-# Needed for Yarn client mode
---conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/libmyudf.so
-```
-
-- Use `--archives`
-```shell
---archives /path/to/udf_archives.zip#udf_archives
---conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=udf_archives
-# Needed for Yarn client mode
---conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/udf_archives.zip
-```
-
-- Specify URI
-
-You can also specify the local or HDFS URIs to the UDF libraries or archives. Local URIs should exist on driver and every worker nodes.
-```shell
---conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=hdfs://path/to/library_or_archive
-```
-
-## Try the example
-
-We provided an Velox UDF example file [MyUDF.cpp](../../cpp/velox/udf/examples/MyUDF.cpp). After building gluten cpp, you can find the example library at /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
-
-Start spark-shell or spark-sql with below configuration
-```
---files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
---conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
-```
-Run query. The functions `myudf1` and `myudf2` increment the input value by a constant of 5
-```
-select myudf1(1), myudf2(100L)
-```
-The output from spark-shell will be like
-```
-+----------------+------------------+
-|udfexpression(1)|udfexpression(100)|
-+----------------+------------------+
-| 6| 105|
-+----------------+------------------+
-```
+Please check the [VeloxNativeUDF.md](../developers/VeloxNativeUDF.md) for more detailed usage and configurations.
# High-Bandwidth Memory (HBM) support
@@ -499,7 +417,7 @@ After the set-up, you can now build Gluten with HBM. Below command is used to en
```bash
cd /path/to/gluten
-## The script builds two jars for spark 3.2.2 and 3.3.1.
+## The script builds four jars for spark 3.2.2, 3.3.1, 3.4.2 and 3.5.1.
./dev/buildbundle-veloxbe.sh --enable_hbm=ON
```
@@ -514,6 +432,7 @@ Gluten supports using Intel® QuickAssist Technology (QAT) for data compression
This feature is based on QAT driver library and [QATzip](https://github.com/intel/QATzip) library. Please manually download QAT driver for your system, and follow its README to build and install on all Driver and Worker node: [Intel® QuickAssist Technology Driver for Linux* – HW Version 2.0](https://www.intel.com/content/www/us/en/download/765501/intel-quickassist-technology-driver-for-linux-hw-version-2-0.html?wapkw=quickassist).
## Software Requirements
+
- Download QAT driver for your system, and follow its README to build and install on all Driver and Worker nodes: [Intel® QuickAssist Technology Driver for Linux* – HW Version 2.0](https://www.intel.com/content/www/us/en/download/765501/intel-quickassist-technology-driver-for-linux-hw-version-2-0.html?wapkw=quickassist).
- Below compression libraries need to be installed on all Driver and Worker nodes:
- Zlib* library of version 1.2.7 or higher
@@ -522,9 +441,7 @@ This feature is based on QAT driver library and [QATzip](https://github.com/inte
## Build Gluten with QAT
-1. Setup ICP_ROOT environment variable to the directory where QAT driver is extracted.
-This environment variable is required during building Gluten and running Spark applications.
-It's recommended to put it in .bashrc on Driver and Worker nodes.
+1. Setup ICP_ROOT environment variable to the directory where QAT driver is extracted. This environment variable is required during building Gluten and running Spark applications. It's recommended to put it in .bashrc on Driver and Worker nodes.
```bash
echo "export ICP_ROOT=/path/to/QAT_driver" >> ~/.bashrc
@@ -536,9 +453,8 @@ echo "export ICP_ROOT=/path/to/QAT_driver" >> ~/.bashrc
exit
```
-2. **This step is required if your application is running as Non-root user**.
-The users must be added to the 'qat' group after QAT drvier is installed.
-And change the amount of max locked memory for the username that is included in the group name. This can be done by specifying the limit in /etc/security/limits.conf.
+2. **This step is required if your application is running as Non-root user.**
+ The users must be added to the 'qat' group after QAT drvier is installed. And change the amount of max locked memory for the username that is included in the group name. This can be done by specifying the limit in /etc/security/limits.conf.
```bash
sudo su -
@@ -550,9 +466,7 @@ echo "@qat - memlock 500000" >> /etc/security/limits.conf
exit
```
-3. Enable huge page. This step is required to execute each time after system reboot. We recommend using systemctl to manage at system startup.
-You change the values for "max_huge_pages" and "max_huge_pages_per_process" to make sure there are enough resources for your workload.
-As for Spark applications, one process matches one executor. Within the executor, every task is allocated a maximum of 5 huge pages.
+3. Enable huge page. This step is required to execute each time after system reboot. We recommend using systemctl to manage at system startup. You change the values for "max_huge_pages" and "max_huge_pages_per_process" to make sure there are enough resources for your workload. As for Spark applications, one process matches one executor. Within the executor, every task is allocated a maximum of 5 huge pages.
```bash
sudo su -
@@ -589,7 +503,7 @@ exit
```bash
cd /path/to/gluten
-## The script builds two jars for spark 3.2.2 and 3.3.1.
+## The script builds four jars for spark 3.2.2, 3.3.1, 3.4.2 and 3.5.1.
./dev/buildbundle-veloxbe.sh --enable_qat=ON
```
@@ -634,7 +548,7 @@ There is 8 QAT acceleration device(s) in the system:
--conf spark.gluten.sql.columnar.shuffle.codecBackend=qat
```
-4. You can use below command to check whether QAT is working normally at run-time. The value of fw_counters should continue to increase during shuffle.
+4. You can use below command to check whether QAT is working normally at run-time. The value of fw_counters should continue to increase during shuffle.
```
while :; do cat /sys/kernel/debug/qat_4xxx_0000:6b:00.0/fw_counters; sleep 1; done
@@ -670,7 +584,8 @@ This feature is based on Intel® [QPL](https://github.com/intel/qpl).
Gluten will internally build and link to a specific version of QPL library, but extra environment setup is still required. Please refer to [QPL Installation Guide](https://intel.github.io/qpl/documentation/get_started_docs/installation.html) to install dependencies and configure accelerators.
-**This step is required if your application is running as Non-root user**. Create a group for the users who have privilege to use IAA, and grant group iaa read/write access to the IAA Work-Queues.
+**This step is required if your application is running as Non-root user.**
+Create a group for the users who have privilege to use IAA, and grant group iaa read/write access to the IAA Work-Queues.
```bash
sudo groupadd iaa
@@ -678,13 +593,13 @@ sudo usermod -aG iaa username # need to relogin
sudo chgrp -R iaa /dev/iax
sudo chmod -R g+rw /dev/iax
```
-
+
After the set-up, you can now build Gluten with QAT. Below command is used to enable this feature
```bash
cd /path/to/gluten
-## The script builds two jars for spark 3.2.2 and 3.3.1.
+## The script builds four jars for spark 3.2.2, 3.3.1, 3.4.2 and 3.5.1.
./dev/buildbundle-veloxbe.sh --enable_iaa=ON
```
@@ -697,6 +612,7 @@ sudo ls -l /dev/iax
```
The output should be like:
+
```
total 0
crw-rw---- 1 root iaa 509, 0 Apr 5 18:54 wq1.0
@@ -728,7 +644,7 @@ Check out the [Intel® Query Processing Library (Intel® QPL) Documentation](htt
# Test TPC-H or TPC-DS on Gluten with Velox backend
-All TPC-H and TPC-DS queries are supported in Gluten Velox backend.
+All TPC-H and TPC-DS queries are supported in Gluten Velox backend.
## Data preparation
@@ -739,6 +655,7 @@ The used TPC-H and TPC-DS queries are the original ones, and can be accessed fro
and [TPC-H queries](../../gluten-core/src/test/resources/tpch-queries).
Some other versions of TPC-DS queries are also provided, but are **not** recommended for testing, including:
+
- the modified TPC-DS queries with "Decimal-to-Double": [TPC-DS non-decimal queries](../../gluten-core/src/test/resources/tpcds-queries/tpcds.queries.no-decimal) (outdated).
## Submit the Spark SQL job
@@ -751,13 +668,13 @@ var parquet_file_path = "/PATH/TO/TPCH_PARQUET_PATH"
var gluten_root = "/PATH/TO/GLUTEN"
```
-Below script shows an example about how to run the testing, you should modify the parameters such as executor cores, memory, offHeap size based on your environment.
+Below script shows an example about how to run the testing, you should modify the parameters such as executor cores, memory, offHeap size based on your environment.
```bash
export GLUTEN_JAR = /PATH/TO/GLUTEN/package/target/<gluten-jar>
cat tpch_parquet.scala | spark-shell --name tpch_powertest_velox \
--master yarn --deploy-mode client \
- --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.driver.extraClassPath=${GLUTEN_JAR} \
--conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.memory.offHeap.enabled=true \
@@ -775,7 +692,8 @@ cat tpch_parquet.scala | spark-shell --name tpch_powertest_velox \
Refer to [Gluten configuration](../Configuration.md) for more details.
## Result
-*wholestagetransformer* indicates that the offload works.
+
+*wholestagetransformer* indicates that the offloading works.
@@ -904,7 +822,7 @@ ashProbe: Input: 9 rows (864B, 3 batches), Output: 27 rows (3.56KB, 3 batches),
# Gluten Implicits
-Gluten provides a helper class to get the fallback summary from a Spark Dataset.
+Gluten provides a helper class to get the fallback summary from a Spark Dataset.
```
import org.apache.spark.sql.execution.GlutenImplicits._
diff --git a/docs/get-started/VeloxABFS.md b/docs/get-started/VeloxABFS.md
new file mode 100644
index 0000000..1aa4ec8
--- /dev/null
+++ b/docs/get-started/VeloxABFS.md
@@ -0,0 +1,25 @@
+---
+layout: page
+title: Using ABFS with Gluten
+nav_order: 6
+parent: Getting-Started
+grand_parent: Documentations
+permalink: /docs/getting-started/abfs/
+---
+ABFS is an important data store for big data users. This doc discusses config details and use cases of Gluten with ABFS. To use an ABFS account as your data source, please ensure you use the listed ABFS config in your spark-defaults.conf. If you would like to authenticate with ABFS using additional auth mechanisms, please reach out using the 'Issues' tab.
+
+# Working with ABFS
+
+## Configuring ABFS Access Token
+
+To configure access to your storage account, replace <storage-account> with the name of your account. This property aligns with Spark configurations. By setting this config multiple times using different storage account names, you can access multiple ABFS accounts.
+
+```sh
+spark.hadoop.fs.azure.account.key.<storage-account>.dfs.core.windows.net XXXXXXXXX
+```
+
+### Other authentatication methods are not yet supported.
+
+# Local Caching support
+
+Velox supports a local cache when reading data from ABFS. Please refer [Velox Local Cache](VeloxLocalCache.md) part for more detailed configurations.
diff --git a/docs/VeloxGCS.md b/docs/get-started/VeloxGCS.md
similarity index 72%
rename from docs/VeloxGCS.md
rename to docs/get-started/VeloxGCS.md
index 73318e9..3fd362d 100644
--- a/docs/VeloxGCS.md
+++ b/docs/get-started/VeloxGCS.md
@@ -1,10 +1,10 @@
---
layout: page
title: Using GCS with Gluten
-nav_order: 4
-parent: Velox Backend
-grand_parent: Documentation
-permalink: /docs/velox/gcs
+nav_order: 5
+parent: Getting-Started
+grand_parent: Documentations
+permalink: /docs/getting-started/gcs/
---
Object stores offered by CSPs such as GCS are important for users of Gluten to store their data. This doc will discuss all details of configs, and use cases around using Gluten with object stores. In order to use a GCS endpoint as your data source, please ensure you are using the following GCS configs in your spark-defaults.conf. If you're experiencing any issues authenticating to GCS with additional auth mechanisms, please reach out to us using the 'Issues' tab.
@@ -39,3 +39,17 @@ For cases when a GCS mock is used, an optional endpoint can be provided:
```sh
spark.hadoop.fs.gs.storage.root.url // url to the mock gcs service including starting with http or https
```
+
+## Configuring GCS max retry count
+
+For cases when a transient server error is detected, GCS can be configured to keep retrying until a number of transient error is detected.
+```sh
+spark.hadoop.fs.gs.http.max.retry // number of times to keep retrying unless a non-transient error is detected
+```
+
+## Configuring GCS max retry time
+
+For cases when a transient server error is detected, GCS can be configured to keep retrying until the retry loop exceeds a prescribed duration.
+```sh
+spark.hadoop.fs.gs.http.max.retry-time // a string representing the time keep retring (10s, 1m, etc).
+```
diff --git a/docs/get-started/VeloxLocalCache.md b/docs/get-started/VeloxLocalCache.md
new file mode 100644
index 0000000..ead04ed
--- /dev/null
+++ b/docs/get-started/VeloxLocalCache.md
@@ -0,0 +1,22 @@
+---
+layout: page
+title: Velox Local Caching
+nav_order: 7
+parent: Getting-Started
+grand-parent: Documentations
+permalink: /docs/getting-started/localcache/
+---
+
+Velox supports a local cache when reading data from HDFS/S3/ABFS. With this feature, Velox can asynchronously cache the data on local disk when reading from remote storage and future read requests on previously cached blocks will be serviced from local cache files. To enable the local caching feature, the following configurations are required:
+
+```
+spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable velox cache, default false.
+spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of in-mem cache, default is 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to store the cache files, default is "/tmp".
+spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of the SSD cache, default is 128MB. Velox will do in-mem cache only if this value is 0.
+spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the SSD cache, default is 1.
+spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the IO threads for cache promoting, default is 1. Velox will try to do "read-ahead" if this value is bigger than 1
+spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable O_DIRECT on cache write, default false.
+```
+
+It's recommended to mount SSDs to the cache path to get the best performance of local caching. Cache files will be written to "spark.gluten.sql.columnar.backend.velox.cachePath", with UUID based suffix, e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten cannot reuse older caches for now, and the old cache files are left after Spark context shutdown.
diff --git a/docs/VeloxS3.md b/docs/get-started/VeloxS3.md
similarity index 55%
rename from docs/VeloxS3.md
rename to docs/get-started/VeloxS3.md
index 3a1a5f7..741b4aa 100644
--- a/docs/VeloxS3.md
+++ b/docs/get-started/VeloxS3.md
@@ -1,10 +1,10 @@
---
layout: page
title: Using S3 with Gluten
-nav_order: 3
-parent: Velox Backend
-grand_parent: Documentation
-permalink: /docs/velox/s3
+nav_order: 4
+parent: Getting-Started
+grand_parent: Documentations
+permalink: /docs/getting-started/s3/
---
Object stores offered by CSPs such as AWS S3 are important for users of Gluten to store their data. This doc will discuss all details of configs, and use cases around using Gluten with object stores. In order to use an S3 endpoint as your data source, please ensure you are using the following S3 configs in your spark-defaults.conf. If you're experiencing any issues authenticating to S3 with additional auth mechanisms, please reach out to us using the 'Issues' tab.
@@ -60,16 +60,4 @@ You can change log granularity of AWS C++ SDK by setting the `spark.gluten.velox
# Local Caching support
-Velox supports a local cache when reading data from HDFS/S3. The feature is very useful if remote storage is slow, e.g., reading from a public S3 bucket and stronger performance is desired. With this feature, Velox can asynchronously cache the data on local disk when reading from remote storage, and the future reading requests on already cached blocks will be serviced from local cache files. To enable the local caching feature, below configurations are required:
-
-```
-spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable velox cache, default false.
-spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of in-mem cache, default is 128MB.
-spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to store the cache files, default is "/tmp".
-spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of the SSD cache, default is 128MB. Velox will do in-mem cache only if this value is 0.
-spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the SSD cache, default is 1.
-spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the IO threads for cache promoting, default is 1. Velox will try to do "read-ahead" if this value is bigger than 1
-spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable O_DIRECT on cache write, default false.
-```
-
-It's recommended to mount SSDs to the cache path to get the best performance of local caching. On the start up of Spark context, the cache files will be allocated under "spark.gluten.sql.columnar.backend.velox.cachePath", with UUID based suffix, e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten is not able to reuse older caches for now, and the old cache files are left there after Spark context shutdown.
+Velox supports a local cache when reading data from S3. Please refer [Velox Local Cache](VeloxLocalCache.md) part for more detailed configurations.
diff --git a/docs/get-started/Work-with-pyspark.ipynb b/docs/get-started/Work-with-pyspark.ipynb
new file mode 100644
index 0000000..23d7c77
--- /dev/null
+++ b/docs/get-started/Work-with-pyspark.ipynb
@@ -0,0 +1,136 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "509d5a1e-10f2-4294-a104-9fdb85a29b0c",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "SLF4J: Class path contains multiple SLF4J bindings.\n",
+ "SLF4J: Found binding in [jar:file:/root/spark-3.3.1-bin-hadoop2/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\n",
+ "SLF4J: Found binding in [jar:file:/root/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\n",
+ "SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\n",
+ "SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]\n",
+ "24/04/03 22:04:56 WARN Utils: Your hostname, vsr542 resolves to a loopback address: 127.0.1.1; using 10.0.2.142 instead (on interface eno1)\n",
+ "24/04/03 22:04:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
+ "Setting default log level to \"WARN\".\n",
+ "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
+ "24/04/03 22:04:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
+ "24/04/03 22:05:00 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import findspark\n",
+ "findspark.init()\n",
+ "\n",
+ "from pyspark import SparkConf, SparkContext\n",
+ "nativesql_jars = \"/path/to/gluten-XXXX.jar\"\n",
+ "conf = SparkConf().setAppName(\"PySpark Gluten\").setMaster(\"yarn\")\n",
+ "conf.set(\"spark.executor.instances\", \"2\")\n",
+ "conf.set(\"spark.executor.memory\", \"6g\")\n",
+ "conf.set(\"spark.executor.cores\", \"2\")\n",
+ "conf.set(\"spark.driver.memory\", \"2g\")\n",
+ "conf.set(\"spark.memory.offHeap.enabled\", \"true\")\n",
+ "conf.set(\"spark.memory.offHeap.size\", \"2g\")\n",
+ "conf.set(\"spark.executor.memoryOverhead\", \"384M\")\n",
+ "conf.set(\"spark.driver.extraClassPath\", nativesql_jars)\n",
+ "conf.set(\"spark.executor.extraClassPath\", nativesql_jars)\n",
+ "conf.set(\"spark.plugins\", \"org.apache.gluten.GlutenPlugin\")\n",
+ "conf.set(\"spark.gluten.loadLibFromJar\", \"false\")\n",
+ "conf.set(\"spark.shuffle.manager\", \"org.apache.spark.shuffle.sort.ColumnarShuffleManager\")\n",
+ "sc = SparkContext(conf=conf)\n",
+ "from pyspark.sql import SparkSession\n",
+ "spark_session = SparkSession(sc)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "6c4ec66a-3b8c-4a65-b948-8420b492333d",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "== Physical Plan ==\n",
+ "VeloxColumnarToRowExec\n",
+ "+- ^(1) FilterExecTransformer (isnotnull(category#1) AND (category#1 = Cellphone))\n",
+ " +- ^(1) InputIteratorTransformer[product#0, category#1, revenue#2L]\n",
+ " +- ^(1) InputAdapter\n",
+ " +- ^(1) RowToVeloxColumnar\n",
+ " +- *(1) Scan ExistingRDD[product#0,category#1,revenue#2L]\n",
+ "\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "df = spark_session.createDataFrame(\n",
+ " [\n",
+ " (\"Normal\", \"Cellphone\", 6000),\n",
+ " (\"Normal\", \"Tablet\", 1500),\n",
+ " (\"Mini\", \"Tablet\", 5500),\n",
+ " (\"Mini\", \"Cellphone\", 5000),\n",
+ " (\"Foldable\", \"Cellphone\", 6500),\n",
+ " (\"Foldable\", \"Tablet\", 2500),\n",
+ " (\"Pro\", \"Cellphone\", 3000),\n",
+ " (\"Pro\", \"Tablet\", 4000),\n",
+ " (\"Pro Max\", \"Cellphone\", 4500)\n",
+ " ],\n",
+ " [\"product\", \"category\", \"revenue\"]\n",
+ ")\n",
+ "df.filter(\"category = 'Cellphone'\").explain()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.19"
+ },
+ "nbTranslate": {
+ "displayLangs": [
+ "*"
+ ],
+ "hotkey": "alt-t",
+ "langInMainMenu": true,
+ "sourceLang": "en",
+ "targetLang": "fr",
+ "useGoogleTranslate": true
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/get-started/build-guide.md b/docs/get-started/build-guide.md
new file mode 100644
index 0000000..bc06706
--- /dev/null
+++ b/docs/get-started/build-guide.md
@@ -0,0 +1,81 @@
+---
+layout: page
+title: Build Parameters for Velox Backend
+nav_order: 3
+parent: Getting-Started
+grand_parent: Documentations
+permalink: /docs/getting-started/build-guide/
+---
+## Build Parameters
+### Native build parameters for buildbundle-veloxbe.sh or builddeps-veloxbe.sh
+Please set them via `--`, e.g. `--build_type=Release`.
+
+| Parameters | Description | Default |
+|------------------------|----------------------------------------------------------------------------------------------------|---------|
+| build_type | Build type for Velox & gluten cpp, CMAKE_BUILD_TYPE. | Release |
+| build_tests | Build gluten cpp tests. | OFF |
+| build_examples | Build udf example. | OFF |
+| build_benchmarks | Build gluten cpp benchmarks. | OFF |
+| build_jemalloc | Build with jemalloc. | OFF |
+| build_protobuf | Build protobuf lib. | OFF |
+| enable_qat | Enable QAT for shuffle data de/compression. | OFF |
+| enable_iaa | Enable IAA for shuffle data de/compression. | OFF |
+| enable_hbm | Enable HBM allocator. | OFF |
+| enable_s3 | Build with S3 support. | OFF |
+| enable_gcs | Build with GCS support. | OFF |
+| enable_hdfs | Build with HDFS support. | OFF |
+| enable_abfs | Build with ABFS support. | OFF |
+| enable_ep_cache | Enable caching for external project build (Velox). | OFF |
+| enable_vcpkg | Enable vcpkg for static build. | OFF |
+| run_setup_script | Run setup script to install Velox dependencies. | ON |
+| velox_repo | Specify your own Velox repo to build. | "" |
+| velox_branch | Specify your own Velox branch to build. | "" |
+| velox_home | Specify your own Velox source path to build. | "" |
+| build_velox_tests | Build Velox tests. | OFF |
+| build_velox_benchmarks | Build Velox benchmarks (velox_tests and connectors will be disabled if ON) | OFF |
+| build_arrow | Build arrow java/cpp and install the libs in local. Can turn it OFF after first build. | ON |
+| spark_version | Build for specified version of Spark(3.2, 3.3, 3.4, 3.5, ALL). `ALL` means build for all versions. | ALL |
+
+### Velox build parameters for build_velox.sh
+Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
+
+| Parameters | Description | Default |
+|------------------|---------------------------------------------------------------|------------------------------------------|
+| velox_home | Specify Velox source path to build. | GLUTEN_SRC/ep/build-velox/build/velox_ep |
+| build_type | Velox build type, i.e., CMAKE_BUILD_TYPE. | Release |
+| enable_s3 | Build Velox with S3 support. | OFF |
+| enable_gcs | Build Velox with GCS support. | OFF |
+| enable_hdfs | Build Velox with HDFS support. | OFF |
+| enable_abfs | Build Velox with ABFS support. | OFF |
+| run_setup_script | Run setup script to install Velox dependencies before build. | ON |
+| enable_ep_cache | Enable and reuse cache of Velox build. | OFF |
+| build_test_utils | Build Velox with cmake arg -DVELOX_BUILD_TEST_UTILS=ON if ON. | OFF |
+| build_tests | Build Velox test. | OFF |
+| build_benchmarks | Build Velox benchmarks. | OFF |
+
+### Maven build parameters
+The below parameters can be set via `-P` for mvn.
+
+| Parameters | Description | Default state |
+|---------------------|---------------------------------------|---------------|
+| backends-velox | Build Gluten Velox backend. | disabled |
+| backends-clickhouse | Build Gluten ClickHouse backend. | disabled |
+| celeborn | Build Gluten with Celeborn. | disabled |
+| uniffle | Build Gluten with Uniffle. | disabled |
+| delta | Build Gluten with Delta Lake support. | disabled |
+| iceberg | Build Gluten with Iceberg support. | disabled |
+| spark-3.2 | Build Gluten for Spark 3.2. | enabled |
+| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
+| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
+| spark-3.5 | Build Gluten for Spark 3.5. | disabled |
+
+## Gluten Jar for Deployment
+The gluten jar built out is under `GLUTEN_SRC/package/target/`.
+It's name pattern is `gluten-<backend_type>-bundle-spark<spark.bundle.version>_<scala.binary.version>-<os.detected.release>_<os.detected.release.version>-<project.version>.jar`.
+
+| Spark Version | spark.bundle.version | scala.binary.version |
+|---------------|----------------------|----------------------|
+| 3.2.2 | 3.2 | 2.12 |
+| 3.3.1 | 3.3 | 2.12 |
+| 3.4.2 | 3.4 | 2.12 |
+| 3.5.1 | 3.5 | 2.12 |
diff --git a/docs/get-started/index.md b/docs/get-started/index.md
new file mode 100644
index 0000000..dbb198a
--- /dev/null
+++ b/docs/get-started/index.md
@@ -0,0 +1,11 @@
+---
+layout: page
+title: Getting-Started
+nav_order: 1
+has_children: true
+parent: Documentations
+permalink: /docs/getting-started/
+---
+# Getting Started with Gluten for Apache Spark
+
+
diff --git a/docs/index.md b/docs/index.md
index 234d8ab..f38313d 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,6 +1,6 @@
---
layout: page
-title: Documentation
+title: Documentations
nav_order: 2
has_children: true
permalink: /docs/
diff --git a/docs/velox-backend/index.md b/docs/velox-backend/index.md
new file mode 100644
index 0000000..fa33f10
--- /dev/null
+++ b/docs/velox-backend/index.md
@@ -0,0 +1,11 @@
+---
+layout: page
+title: Velox Backend
+nav_order: 3
+has_children: true
+parent: Documentations
+permalink: /docs/velox-backend/
+---
+# Gluten Documents by version
+
+
diff --git a/docs/velox-backend/velox-backend-limitations.md b/docs/velox-backend/velox-backend-limitations.md
new file mode 100644
index 0000000..eb44c91
--- /dev/null
+++ b/docs/velox-backend/velox-backend-limitations.md
@@ -0,0 +1,168 @@
+---
+layout: page
+title: Limitations
+nav_order: 1
+parent: Velox Backend
+grand_parent: Documentations
+permalink: /docs/velox-backend/limitations/
+---
+This document describes the limitations of velox backend by listing some known cases where exception will be thrown, gluten behaves incompatibly with spark, or certain plan's execution
+must fall back to vanilla spark, etc.
+
+### Override of Spark classes (For Spark3.2 and Spark3.3)
+Gluten avoids to modify Spark's existing code and use Spark APIs if possible. However, some APIs aren't exposed in Vanilla spark and we have to copy the Spark file and do the hardcode changes. The list of override classes can be found as ignoreClasses in package/pom.xml . If you use customized Spark, you may check if the files are modified in your spark, otherwise your changes will be overrided.
+
+So you need to ensure preferentially load the Gluten jar to overwrite the jar of vanilla spark. Refer to [How to prioritize loading Gluten jars in Spark](https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-troubleshooting.md#incompatible-class-error-when-using-native-writer).
+
+If not officially supported spark3.2/3.3 version is used, NoSuchMethodError can be thrown at runtime. More details see [issue-4514](https://github.com/apache/incubator-gluten/issues/4514).
+
+### Fallbacks
+Except the unsupported operators, functions, file formats, data sources listed in , there are some known cases also fall back to Vanilla Spark.
+
+#### ANSI
+Gluten currently doesn't support ANSI mode. If ANSI is enabled, Spark plan's execution will always fall back to vanilla Spark.
+
+#### Runtime BloomFilter
+Velox BloomFilter's serialization format is different from Spark's. BloomFilter binary generated by Velox can't be deserialized by vanilla spark. So if `might_contain` falls back, we fall back `bloom_filter_agg` to vanilla spark also.
+
+#### Case Sensitive mode
+Gluten only supports spark default case-insensitive mode. If case-sensitive mode is enabled, user may get incorrect result.
+
+#### Regexp functions
+In Velox, regexp functions (`rlike`, `regexp_extract`, etc.) are implemented based on RE2, while in Spark they are based on `java.util.regex`.
+* Lookaround (lookahead/lookbehind) pattern is not supported in RE2.
+* When matching white space with pattern "\\s", RE2 doesn't treat "\v" (or "\x0b") as white space, but `java.util.regex` does.
+
+There are a few unknown incompatible cases. If user cannot tolerate the incompatibility risk, please enable the below configuration property.
+```
+spark.gluten.sql.fallbackRegexpExpressions
+```
+
+#### FileSource format
+Currently, Gluten only fully supports parquet file format and partially support ORC. If other format is used, scan operator falls back to vanilla spark.
+
+#### Partitioned Table Scan
+Gluten only support the partitioned table scan when the file path contain the partition info, otherwise will fall back to vanilla spark.
+
+### Incompatible behavior
+In certain cases, Gluten result may be different from Vanilla spark.
+
+#### JSON functions
+Velox only supports double quotes surrounded strings, not single quotes, in JSON data. If single quotes are used, gluten will produce incorrect result.
+
+Velox doesn't support [*] in path when get_json_object function is called and returns null instead.
+
+#### Parquet read conf
+Gluten supports `spark.files.ignoreCorruptFiles` with default false, if true, the behavior is same as config false.
+Gluten ignores `spark.sql.parquet.datetimeRebaseModeInRead`, it only returns what write in parquet file. It does not consider the difference between legacy
+hybrid (Julian Gregorian) calendar and Proleptic Gregorian calendar. The result may be different with vanilla spark.
+
+#### Parquet write conf
+Spark has `spark.sql.parquet.datetimeRebaseModeInWrite` config to decide whether legacy hybrid (Julian + Gregorian) calendar
+or Proleptic Gregorian calendar should be used during parquet writing for dates/timestamps. If the parquet to read is written
+by Spark with this config as true, Velox's TableScan will output different result when reading it back.
+
+#### Partition write (For Spark3.2 and Spark3.3)
+
+Gluten only supports static partition writes and does not support dynamic partition writes.
+
+```scala
+spark.sql("CREATE TABLE t (c int, d long, e long) STORED AS PARQUET partitioned by (c, d)")
+spark.sql("INSERT OVERWRITE TABLE t partition(c=1, d=2) SELECT 3 as e")
+```
+Gluten does not support dynamic partition write and bucket write, Exception may be raised if you use. e.g.,
+
+```scala
+spark.range(100).selectExpr("id as c1", "id % 7 as p")
+ .write
+ .format("parquet")
+ .partitionBy("p")
+ .save(f.getCanonicalPath)
+```
+
+#### Partition write (For Spark3.4 and later)
+
+Gluten supports static partition writes and dynamic partition writes.
+
+```scala
+spark.sql("CREATE TABLE t (c int, d long, e long) STORED AS PARQUET partitioned by (c, d)")
+spark.sql("INSERT OVERWRITE TABLE t partition(c=1, d) SELECT 2 as d, 3 as e")
+```
+
+Gluten does not support bucket write, and will fall back to vanilla Spark.
+
+```scala
+spark.range(100).selectExpr("id as c1", "id % 7 as p")
+ .write
+ .format("parquet")
+ .bucketBy(2, "c1")
+ .save(f.getCanonicalPath)
+```
+
+#### CTAS write (For Spark3.2 and Spark3.3)
+
+Gluten does not create table as select. It may raise exception. e.g.,
+
+```scala
+spark.range(100).toDF("id")
+ .write
+ .format("parquet")
+ .saveAsTable("velox_ctas")
+```
+
+#### CTAS write (For Spark3.4 and later)
+
+Gluten supports create table as select with parquet file format.
+
+```scala
+spark.range(100).toDF("id")
+ .write
+ .format("parquet")
+ .saveAsTable("velox_ctas")
+```
+
+#### HiveFileFormat write
+
+Gluten supports writes of HiveFileFormat when the output file type is of type `parquet` only
+
+#### NaN support
+Velox does NOT support NaN. So unexpected result can be obtained for a few cases, e.g., comparing a number with NaN.
+
+#### Configuration
+
+Parquet write only support three configs, other will not take effect.
+
+- compression code:
+ - sql conf: `spark.sql.parquet.compression.codec`
+ - option: `compression.codec`
+- block size
+ - sql conf: `spark.gluten.sql.columnar.parquet.write.blockSize`
+ - option: `parquet.block.size`
+- block rows
+ - sql conf: `spark.gluten.sql.native.parquet.write.blockRows`
+ - option: `parquet.block.rows`
+
+
+
+### Fetal error caused by Spark's columnar reading
+If the user enables Spark's columnar reading, error can occur due to Spark's columnar vector is not compatible with
+Gluten's.
+
+### Spill
+
+`OutOfMemoryExcetpion` may still be triggered within current implementation of spill-to-disk feature, when shuffle partitions is set to a large number. When this case happens, please try to reduce the partition number to get rid of the OOM.
+
+### Unsupported Data type support in ParquetScan
+
+- Byte type causes fallback to vanilla spark
+- Timestamp type
+
+ Only reading with INT96 and dictionary encoding is supported. When reading INT64 represented millisecond/microsecond timestamps, or INT96 represented timestamps of other encodings, exceptions can occur.
+
+- Complex types
+ - Parquet scan of nested array with struct or array as element type is not supported in Velox (fallback behavior).
+ - Parquet scan of nested map with struct as key type, or array type as value type is not supported in Velox (fallback behavior).
+
+### CSV Read
+The header option should be true. And now we only support DatasourceV1, i.e., user should set `spark.sql.sources.useV1SourceList=csv`. User defined read option is not supported, which will make CSV read fall back to vanilla Spark in most case.
+CSV read will also fall back to vanilla Spark and log warning when user specifies schema is different with file schema.
diff --git a/docs/velox-backend/velox-backend-support-progress.md b/docs/velox-backend/velox-backend-support-progress.md
new file mode 100644
index 0000000..36dd75f
--- /dev/null
+++ b/docs/velox-backend/velox-backend-support-progress.md
@@ -0,0 +1,461 @@
+---
+layout: page
+title: Supported Operators & Functions
+nav_order: 2
+parent: Velox Backend
+grand_parent: Documentations
+permalink: /docs/velox-backend/support/
+---
+# The Operators and Functions Support Progress
+
+Gluten is still under active development. Here is a list of supported operators and functions.
+
+Since the same function may have different semantics between Presto and Spark, Velox implement the functions in Presto category, if we note a
+different semantics from Spark, then the function is implemented in Spark category. So Gluten will first try to find function in Velox's spark
+category, if a function isn't implemented then refer to Presto category.
+
+The total number of functions in [Spark3.3](https://spark.apache.org/docs/latest/api/sql/) is 387, Gluten supports 189 of them.
+
+We use some notations to describe the supporting status of operators/functions in the tables below, they are:
+
+| Value | Description |
+|--------------|-------------------------------------------------------------------------------------------|
+| S | Supported. Gluten or Velox supports fully. |
+| S* | Mark for foldable expression that will be converted to alias after spark's optimization. |
+| [Blank Cell] | Not applicable case or needs to confirm. |
+| PS | Partial Support. Velox only partially supports it. |
+| NS | Not Supported. Velox backend does not support it. |
+
+
+And also some notations for the function implementation's restrictions:
+
+| Value | Description |
+|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mismatched | Some functions are implemented by Velox, but have different semantics from Apache Spark, we mark them as "Mismatched". |
+| ANSI OFF | Gluten doesn't support [ANSI mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html). If it is enabled, Gluten will fall back to Vanilla Spark. |
+
+### Operator Map
+
+Gluten supports 28 operators (Drag to right to see all data types)
+
+| Executor | Description | Gluten Name | Velox Name | BOOLEAN | BYTE | SHORT | INT | LONG | FLOAT | DOUBLE | STRING | NULL | BINARY | ARRAY | MAP | STRUCT(ROW) | DATE | TIMESTAMP | DECIMAL | CALENDAR | UDT |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|-----------------------|---------|------|-------|-----|------|-------|--------|--------|------|--------|-------|-----|-------------|------|-----------|---------|----------|-----|
+| FileSourceScanExec | Reading data from files, often from Hive tables | FileSourceScanExecTransformer | TableScanNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| BatchScanExec | The backend for most file input | BatchScanExecTransformer | TableScanNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| FilterExec | The backend for most filter statements | FilterExecTransformer | FilterNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ProjectExec | The backend for most select, withColumn and dropColumn statements | ProjectExecTransformer | ProjectNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| HashAggregateExec | The backend for hash based aggregations | HashAggregateBaseTransformer | AggregationNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| BroadcastHashJoinExec | Implementation of join using broadcast data | BroadcastHashJoinExecTransformer | HashJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ShuffledHashJoinExec | Implementation of join using hashed shuffled data | ShuffleHashJoinExecTransformer | HashJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| SortExec | The backend for the sort operator | SortExecTransformer | OrderByNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| SortMergeJoinExec | Sort merge join, replacing with shuffled hash join | SortMergeJoinExecTransformer | MergeJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| WindowExec | Window operator backend | WindowExecTransformer | WindowNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| GlobalLimitExec | Limiting of results across partitions | LimitTransformer | LimitNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| LocalLimitExec | Per-partition limiting of results | LimitTransformer | LimitNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ExpandExec | The backend for the expand operator | ExpandExecTransformer | GroupIdNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| UnionExec | The backend for the union operator | UnionExecTransformer | N | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| DataWritingCommandExec | Writing data | Y | TableWriteNode | S | S | S | S | S | S | S | S | S | S | S | NS | S | S | NS | S | NS | NS |
+| CartesianProductExec | Implementation of join using brute force | CartesianProductExecTransformer | NestedLoopJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| ShuffleExchangeExec | The backend for most data being exchanged between processes | ColumnarShuffleExchangeExec | ExchangeNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The unnest operation expands arrays and maps into separate columns | N | UnnestNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The top-n operation reorders a dataset based on one or more identified sort fields as well as a sorting order | N | TopNNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The partitioned output operation redistributes data based on zero or more distribution fields | N | PartitionedOutputNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The values operation returns specified data | N | ValuesNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | A receiving operation that merges multiple ordered streams to maintain orderedness | N | MergeExchangeNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | An operation that merges multiple ordered streams to maintain orderedness | N | LocalMergeNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | Partitions input data into multiple streams or combines data from multiple streams into a single stream | N | LocalPartitionNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The enforce single row operation checks that input contains at most one row and returns that row unmodified | N | EnforceSingleRowNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |
+| | The assign unique id operation adds one column at the end of the input columns with unique value per row | N | AssignUniqueIdNode | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | S | S | S | S | S |
+| ReusedExchangeExec | A wrapper for reused exchange to have different output | ReusedExchangeExec | N | | | | | | | | | | | | | | | | | | |
+| CollectLimitExec | Reduce to single partition and apply limit | N | N | | | | | | | | | | | | | | | | | | |
+| BroadcastExchangeExec | The backend for broadcast exchange of data | Y | Y | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | S | NS | NS |
+| ObjectHashAggregateExec | The backend for hash based aggregations supporting TypedImperativeAggregate functions | N | N | | | | | | | | | | | | | | | | | | |
+| SortAggregateExec | The backend for sort based aggregations | N | N | | | | | | | | | | | | | | | | | | |
+| CoalesceExec | Reduce the partition numbers | CoalesceExecTransformer | N | | | | | | | | | | | | | | | | | | |
+| GenerateExec | The backend for operations that generate more output rows than input rows like explode | GenerateExecTransformer | UnnestNode | | | | | | | | | | | | | | | | | | |
+| RangeExec | The backend for range operator | N | N | | | | | | | | | | | | | | | | | | |
+| SampleExec | The backend for the sample operator | N | N | | | | | | | | | | | | | | | | | | |
+| SubqueryBroadcastExec | Plan to collect and transform the broadcast key values | Y | Y | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | S | NS | NS |
+| TakeOrderedAndProjectExec | Take the first limit elements as defined by the sortOrder, and do projection if needed | Y | Y | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | S | NS | NS |
+| CustomShuffleReaderExec | A wrapper of shuffle query stage | N | N | | | | | | | | | | | | | | | | | | |
+| InMemoryTableScanExec | Implementation of InMemory Table Scan | Y | Y | | | | | | | | | | | | | | | | | | |
+| BroadcastNestedLoopJoinExec | Implementation of join using brute force. Full outer joins and joins where the broadcast side matches the join side (e.g.: LeftOuter with left broadcast) are not supported | BroadcastNestedLoopJoinExecTransformer | NestedLoopJoinNode | S | S | S | S | S | S | S | S | S | S | NS | NS | NS | S | NS | NS | NS | NS |
+| AggregateInPandasExec | The backend for an Aggregation Pandas UDF, this accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| ArrowEvalPythonExec | The backend of the Scalar Pandas UDFs. Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| FlatMapGroupsInPandasExec | The backend for Flat Map Groups Pandas UDF, Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| MapInPandasExec | The backend for Map Pandas Iterator UDF. Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| WindowInPandasExec | The backend for Window Aggregation Pandas UDF, Accelerates the data transfer between the Java process and the Python process | N | N | | | | | | | | | | | | | | | | | | |
+| HiveTableScanExec | The Hive table scan operator. Column and partition pruning are both handled | Y | Y | | | | | | | | | | | | | | | | | | |
+| InsertIntoHiveTable | Command for writing data out to a Hive table | Y | Y | | | | | | | | | | | | | | | | | | |
+| Velox2Row | Convert Velox format to Row format | Y | Y | S | S | S | S | S | S | S | S | NS | S | NS | NS | NS | S | S | NS | NS | NS |
+| Velox2Arrow | Convert Velox format to Arrow format | Y | Y | S | S | S | S | S | S | S | S | NS | S | S | S | S | S | NS | S | NS | NS |
+
+
+### Function support
+
+Gluten supports 199 functions. (Drag to right to see all data types)
+
+| Spark Functions | Velox/Presto Functions | Velox/Spark functions | Gluten | Restrictions | BOOLEAN | BYTE | SHORT | INT | LONG | FLOAT | DOUBLE | DATE | TIMESTAMP | STRING | DECIMAL | NULL | BINARY | CALENDAR | ARRAY | MAP | STRUCT | UDT |
+|------------------------------|------------------------|-----------------------|--------|--------------------------|---------|------|-------|-----|------|-------|--------|------|-----------|--------|---------|------|--------|----------|-------|-----|--------|-----|
+| ! | | not | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| != | neq | | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| % | mod | remainder | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| & | bitwise_and | bitwise_and | S | | | | | | | | | | | | | | | | | | | |
+| * | multiply | multiply | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| + | plus | add | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| - | minus | subtract | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| / | divide | divide | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| < | lt | lessthan | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| <= | lte | lessthanorequa | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| <=> | | equalnullsafe | S | | | | | | | | | | | | | | | | | | | |
+| <> | neq | notequalto | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| = | | | S | | | | | | | | | | | | | | | | | | | |
+| == | eq | equalto | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| > | gt | greaterthan | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| >= | gte | greaterthanorequal | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| ^ | bitwise_xor | | S | | | | S | S | S | | | | | | | | | | | | | |
+| | | bitwise_or | bitwise_or | S | | | | S | S | S | | | | | | | | | | | | | |
+| || | | | S | | | | | | | | | | | | | | | | | | | |
+| ~ | bitwise_not | | S | | | | S | S | S | | | | | | | | | | | | | |
+| and | | | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| between | between | between | S | | S | S | S | S | S | S | S | S | | S | | | | | | | | |
+| bit_and | bitwise_and_agg | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| bit_count | bit_count | bit_count | S | | S | S | S | S | S | | | | | | | | | | | | | |
+| bit_get | | bit_get | S | | | S | S | S | S | | | | | | | | | | | | | |
+| bit_or | | | S | | | | | | | | | | | | | | | | | | | |
+| bit_xor | | bit_xor | S | | | | | | | | | | | | | | | | | | | |
+| case | | | S | | | | | | | | | | | | | | | | | | | |
+| div | | | | | | | | | | | | | | | | | | | | | | |
+| getbit | | | | | | | | | | | | | | | | | | | | | | |
+| if | | | S | | | | | | | | | | | | | | | | | | | |
+| ifnull | | | | | | | | | | | | | | | | | | | | | | |
+| in | | in | S | | | | S | S | S | S | S | S | S | S | | | | | | | | |
+| isnan | is_nan | isnan | S | | | | | | | S | S | | | | | | | | | | | |
+| isnotnull | | isnotnull | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| isnull | is_null | isnull | S | | S | S | S | S | S | S | S | | | S | | | | | | | | |
+| mod | mod | remainder | S | | | | | | | | | | | | | | | | | | | |
+| negative | negate | unaryminus | | | | | | | | | | | | | | | | | | | | |
+| not | | not | S | | S | S | S | S | S | S | S | S | | | | | | | | | | |
+| nullif | | | S | | | | | | | | | | | | | | | | | | | |
+| or | | | S | | | | | | | | | | | | | | | | | | | |
+| positive | | | S | | | | | | | | | | | | | | | | | | | |
+| when | | | S | | | | | | | | | | | | | | | | | | | |
+| ascii | | ascii | S | | | | | | | | | | | S | | | | | | | | |
+| base64 | | | | | | | | | | | | | | | | | | | | | | |
+| bin | | bin | | | | | | | | | | | | | | | | | | | | |
+| bit_length | | | S | | | | | | | | | | | | | | | | | | | |
+| btrim | | | S | | | | | | | | | | | | | | | | | | | |
+| char, chr | chr | chr | S | | | | | | | | | | | S | | | | | | | | |
+| char_length/character_length | length | length | S | | | | | | | | | | | S | | | | | | | | |
+| character_lengt/char_length | length | length | S | | | | | | | | | | | S | | | | | | | | |
+| chr, char | chr | chr | S | | | | | | | | | | | S | | | | | | | | |
+| concat | concat | concat | S | | | | | | | | | | | S | | | | | | | | |
+| concat_ws | | | | | | | | | | | | | | | | | | | | | | |
+| contains | | contains | | | | | | | | | | | | | | | | | | | | |
+| decode | | | | | | | | | | | | | | | | | | | | | | |
+| elt | | | | | | | | | | | | | | | | | | | | | | |
+| encode | | | | | | | | | | | | | | | | | | | | | | |
+| endswith | | endsWith | | | | | | | | | | | | | | | | | | | | |
+| find_in_set | | | S | | | | | | | | | | | S | | | | | | | | |
+| format_number | | | | | | | | | | | | | | | | | | | | | | |
+| format_string | | | | | | | | | | | | | | | | | | | | | | |
+| initcap | | | | | | | | | | | | | | | | | | | | | | |
+| instr | | instr | S | | | | | | | | | | | S | | | | | | | | |
+| lcase, lower | lower | lower | S | | | | | | | | | | | S | | | | | | | | |
+| left | | | S | | | | | | | | | | | S | | | | | | | | |
+| length | length | length | S | | | | | | | | | | | S | | | | | | | | |
+| levenshtein | | levenshtein | S | | | | | | | | | | | | | | | | | | | |
+| locate | strpos | | S | Mismatched | | | | | | | | | | S | | | | | | | | |
+| lower | lower | lower | S | | | | | | | | | | | S | | | | | | | | |
+| lpad | lpad | | S | | | | | | | | | | | S | | | | | | | | |
+| ltrim | ltrim | ltrim | S | | | | | | | | | | | S | | | | | | | | |
+| octet_length | | | | | | | | | | | | | | | | | | | | | | |
+| overlay | | overlay | S | | | | | | | | | | | | | | | | | | | |
+| parse_url | | | | | | | | | | | | | | | | | | | | | | |
+| position | strpos | | | | | | | | | | | | | | | | | | | | | |
+| printf | | | | | | | | | | | | | | | | | | | | | | |
+| repeat | | repeat | S | | | | | | | | | | | | | | | | | | | |
+| replace | replace | replace | S | | | | | | | | | | | S | | | | | | | | |
+| reverse | reverse | | S | | | | | | | | | | | S | | | | | | | | |
+| right | | | | | | | | | | | | | | | | | | | | | | |
+| rpad | rpad | | S | | | | | | | | | | | S | | | | | | | | |
+| rtrim | rtrim | rtrim | S | | | | | | | | | | | S | | | | | | | | |
+| sentences | | | | | | | | | | | | | | | | | | | | | | |
+| soundex | | soundex | S | | | | | | | | | | | | | | | | | | | |
+| space | | | | | | | | | | | | | | | | | | | | | | |
+| split | split | split | S | Mismatched | | | | | | | | | | | | | | | | | | |
+| split_part | split_part | | | Mismatched | | | | | | | | | | | | | | | | | | |
+| startswith | | startsWith | | | | | | | | | | | | | | | | | | | | |
+| substr, substring | substr | substring | S | | | | | | | | | | | S | | | | | | | | |
+| substring, substr | substr | substring | S | | | | | | | | | | | S | | | | | | | | |
+| substring_index | | substring_index | S | | | | | | | | | | | | | | | | | | | |
+| translate | | | | | | | | | | | | | | | | | | | | | | |
+| trim | trim | trim | S | | | | | | | | | | | S | | | | | | | | |
+| ucase, upper | upper | upper | S | | | | | | | | | | | S | | | | | | | | |
+| unbase64 | | | | | | | | | | | | | | | | | | | | | | |
+| unhex | | | S | | | | | | | | | | | | | | | | | | | |
+| upper, ucase | upper | upper | S | | | | | | | | | | | S | | | | | | | | |
+| xpath | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_boolean | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_double | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_float | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_int | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_long | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_number | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_short | | | | | | | | | | | | | | | | | | | | | | |
+| xpath_string | | | | | | | | | | | | | | | | | | | | | | |
+| like | like | | S | | | | | | | | | | | S | | | | | | | | |
+| regexp | | rlike | S | Lookaround unsupported | | | | | | | | | | S | | | | | | | | |
+| regexp_extract | regexp_extract | regexp_extract | S | Lookaround unsupported | | | | | | | | | | S | | | | | | | | |
+| regexp_extract_all | regexp_extract_all | | S | Lookaround unsupported | | | | | | | | | | S | | | | | | | | |
+| regexp_like | regexp_like | rlike | S | Lookaround unsupported | | | | | | | | | | S | | | | | | | | |
+| regexp_replace | regexp_replace | | S | Lookaround unsupported | | | | | | | | | | S | | | | | | | | |
+| rlike | | rlike | S | Lookaround unsupported | | | | | | | | | | S | | | | | | | | |
+| abs | abs | abs | S | ANSI OFF | | S | S | S | S | S | S | | | | | | | | | | | |
+| acos | acos | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| acosh | | acosh | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| asin | asin | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| asinh | | asinh | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| atan | atan | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| atan2 | atan2 | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| atanh | | atanh | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| bround | | | | | | | | | | | | | | | | | | | | | | |
+| cbrt | cbrt | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| ceil | ceil | ceil | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| ceiling | ceiling | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| conv | | | | | | | | | | | | | | | | | | | | | | |
+| cos | cos | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| cosh | cosh | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| cot | | | | | | | | | | | | | | | | | | | | | | |
+| degrees | degrees | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| e | e | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| exp | exp | exp | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| expm1 | | | S | | | | | | | | | | | | | | | | | | | |
+| factorial | | | | | | | | | | | | | | | | | | | | | | |
+| floor | floor | floor | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| hex | | | S | | | | | | S | | | | | S | | | S | | | | | |
+| hypot | | | | | | | | | | | | | | | | | | | | | | |
+| ln | ln | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| log | ln | log | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| log10 | log10 | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| log1p | | | | | | | | | | | | | | | | | | | | | | |
+| log2 | log2 | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| pi | pi | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| pmod | | pmod | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| pow, power | pow,power | power | | | | | S | S | S | S | S | | | | | | | | | | | |
+| power, pow | power,pow | power | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| radians | radians | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| rand | rand | rand | S | | | | | | | | | | | | | | | | | | | |
+| rand | rand | rand | | | | | | | | | | | | | | | | | | | | |
+| random | random | | S | | | | | | | | | | | | | | | | | | | |
+| rint | | rint | S | | | | | | | | | | | | | | | | | | | |
+| round | round | round | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| shiftleft | bitwise_left_shift | shiftleft | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| shiftright | bitwise_right_shift | shiftright | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| shiftrightunsigned | | | | | | | | | | | | | | | | | | | | | | |
+| sign, signum | sign | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| signum, sign | sign | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| sin | sin | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| sinh | | sinh | | | | | | | | | | | | | | | | | | | | |
+| sqrt | sqrt | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| tan | tan | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| tanh | tanh | | S | | | S | S | S | S | S | S | | | | | | | | | | | |
+| width_bucket | width_bucket | width_bucket | S | | | | | | | | | | | | | | | | | | | |
+| array | | array | S | | | | | | | | | | | | | | | | S | | | |
+| aggregate | aggregate | reduce | S | | | | | | | | | | | | | | | | S | | | |
+| array_contains | | array_contains | S | | | | | | | | | | | | | | | | | | | |
+| array_distinct | array_distinct | | S | | | | | | | | | | | | | | | | S | | | |
+| array_except | array_except | | S | | | | | | | | | | | | | | | | S | | | |
+| array_intersect | array_intersect | array_intersect | S | | | | | | | | | | | | | | | | | | | |
+| array_join | array_join | | S | | | | | | | | | | | | | | | | | | | |
+| array_max | array_max | | S | | | | | | | | | | | | | | | | | | | |
+| array_min | array_min | | S | | | | | | | | | | | | | | | | | | | |
+| array_position | array_position | | S | | | | | | | | | | | | | | | | S | | | |
+| array_remove | array_remove | | S | | | | | | | | | | | | | | | | | | | |
+| array_repeat | | | S | | S | S | S | S | S | S | S | S | S | S | S | | | | | | | |
+| array_sort | array_sort | array_sort | S | | | | | | | | | | | | | | | | | | | |
+| array_union | | | | | | | | | | | | | | | | | | | | | | |
+| arrays_overlap | array_overlap | S | | | | | | | | | | | | | | | | | | | | |
+| arrays_zip | zip | | S | | | | | | | | | | | | | | | | | | | |
+| cardinality | cardinality | | | | | | | | | | | | | | | | | | | | | |
+| element_at | element_at | element_at | S | | | | | | | | | | | | | | | | S | S | | |
+| exists | any_match | | S | | | | | | | | | | | | | | | | | | | |
+| explode, explode_outer | | | | | | | | | | | | | | | | | | | | | | |
+| explode_outer, explode | | | | | | | | | | | | | | | | | | | | | | |
+| filter | filter | filter | PS | | | | | | | | | | | | | | | | | | | |
+| forall | all_match | | S | | | | | | | | | | | | | | | | | | | |
+| flatten | flatten | flatten | S | | | | | | | | | | | | | | | | | | | |
+| map | map | map | S | | | | | | | | | | | | | | | | | | | |
+| map_concat | map_concat | | | | | | | | | | | | | | | | | | | | | |
+| map_entries | map_entries | | | | | | | | | | | | | | | | | | | | | |
+| map_filter | map_filter | map_filter | | | | | | | | | | | | | | | | | | | | |
+| get_map_value | | element_at | S | | | | | | | | | | | | | | | | | S | | |
+| map_from_arrays | | map_from_arrays | S | | | | | | | | | | | | | | | | | S | | |
+| map_from_entries | map_from_entries | | | | | | | | | | | | | | | | | | | | | |
+| map_keys | map_keys | map_keys | S | | | | | | | | | | | | | | | | | | | |
+| map_values | map_values | map_values | S | | | | | | | | | | | | | | | | | S | | |
+| map_zip_with | map_zip_with | | S | | | | | | | | | | | | | | | | | S | | |
+| named_struct,struct | row_construct | named_struct | S | | | | | | | | | | | | | | | | | | S | |
+| posexplode_outer,posexplode | | | | | | | | | | | | | | | | | | | | | | |
+| sequence | | | | | | | | | | | | | | | | | | | | | | |
+| shuffle | shuffle | shuffle | S | | | | | | | | | | | | | | | | | | | |
+| size | | size | S | | | | | | | | | | | | | | | | | | | |
+| array_size | | | S | | | | | | | | | | | | | | | | | | | |
+| slice | slice | | | | | | | | | | | | | | | | | | | | | |
+| sort_array | | sort_array | S | | | | | | | | | | | | | | | | | | | |
+| str_to_map | | | S | | | | | | | | | | | | | | | | | | | |
+| transform | transform | transofrm | | | | | | | | | | | | | | | | | | | | |
+| transform_keys | transform_keys | | PS | | | | | | | | | | | | | | | | | | | |
+| transform_values | transform_values | | PS | | | | | | | | | | | | | | | | | | | |
+| zip_with | zip_with | | S | | | | | | | | | | | | | | | | | | | |
+| add_months | | | S | | | | | | | | | | | | | | | | | | | |
+| current_date | | | S* | | | | | | | | | | | | | | | | | | | |
+| current_timestamp | | | S* | | | | | | | | | | | | | | | | | | | |
+| current_timezone | | | S* | | | | | | | | | | | | | | | | | | | |
+| date | date | | S | | | | | | | | | | | | | | | | | | | |
+| date_add | date_add | date_add | S | | | S | S | S | | | | S | S | | | | | | | | | |
+| date_format | date_format | | S | | | | | S | | | | | S | | | | | | | | | |
+| date_from_unix_date | | | S | | | | | | | | | | | | | | | | | | | |
+| date_part | | | | | | | | | | | | | | | | | | | | | | |
+| date_sub | | | S | | | | | | | | | | | | | | | | | | | |
+| date_trunc | date_trunc | | | | | | | | | | | | | | | | | | | | | |
+| datediff | date_diff | | S | | | | | | | | | S | S | | | | | | | | | |
+| day | day | | S | | | | | | | | | S | S | | | | | | | | | |
+| dayofmonth | day_of_month | | S | | | | | | | | | S | S | | | | | | | | | |
+| dayofweek | day_of_week,dow | | S | | | | | | | | | S | S | | | | | | | | | |
+| dayofyear | day_of_year,doy | | S | | | | | | | | | S | S | | | | | | | | | |
+| extract | | | | | | | | | | | | S | S | | | | | | | | | |
+| from_unixtime | from_unixtime | | S | | | | | | S | | | | | | | | | | | | | |
+| from_utc_timestamp | | from_utc_timestamp | S | | | | | | | | | | | | | | | | | | | |
+| hour | hour | | S | | | | | | | | | | S | | | | | | | | | |
+| last_day | | last_day | S | | | | | | | | | | | | | | | | | | | |
+| make_date | | make_date | S | | | | | | | | | | | | | | | | | | | |
+| make_dt_interval | | | | | | | | | | | | | | | | | | | | | | |
+| make_interval | | | | | | | | | | | | | | | | | | | | | | |
+| make_timestamp | | | | | | | | | | | | | | | | | | | | | | |
+| make_ym_interval | | | | | | | | | | | | | | | | | | | | | | |
+| minute | minute | | S | | | | | | | | | | | | | | | | | | | |
+| month | month | | S | | | | | | | | | S | S | | | | | | | | | |
+| months_between | | | | | | | | | | | | | | | | | | | | | | |
+| next_day | | | S | | | | | | | | | S | | | | | | | | | | |
+| now | | | S | | | | | | | | | S | S | | | | | | | | | |
+| quarter | quarter | | S | | | | | | | | | S | S | | | | | | | | | |
+| second | second | | S | | | | | | | | | | | | | | | | | | | |
+| session_window | | | | | | | | | | | | | | | | | | | | | | |
+| timestamp | | | | | | | | | | | | | | | | | | | | | | |
+| timestamp_micros | | timestamp_micros | S | | | | | | | | | | | | | | | | | | | |
+| timestamp_millis | | timestamp_millis | S | | | | | | | | | | | | | | | | | | | |
+| timestamp_seconds | | | | | | | | | | | | | | | | | | | | | | |
+| to_date | | | S | | | | | | | | | S | S | | | | | | | | | |
+| to_timestamp | | | | | | | | | | | | | | | | | | | | | | |
+| to_unix_timestamp | to_unixtime | to_unix_timestamp | S | | | | | | | | | | | | | | | | | | | |
+| to_utc_timestamp | | | | | | | | | | | | | | | | | | | | | | |
+| trunc | | | | | | | | | | | | | | | | | | | | | | |
+| unix_timestamp | | unix_timestamp | | | | | | | | | | | | | | | | | | | | |
+| unix_seconds | | unix_seconds | S | | | | | | | | | | | | | | | | | | | |
+| unix_millis | | unix_millis | S | | | | | | | | | | | | | | | | | | | |
+| unix_micros | | unix_micros | S | | | | | | | | | | | | | | | | | | | |
+| weekday | | | S | | | | | | | | | S | | | | | | | | | | |
+| weekofyear | week,week_of_year | | S | | | | | | | | | | | | | | | | | | | |
+| window | | | | | | | | | | | | | | | | | | | | | | |
+| year | year | year | S | | | | | | | | | S | S | | | | | | | | | |
+| aggregate | | aggregate | S | | | | | | | | | | | | | | | | | | | |
+| any | | | | | | | | | | | | | | | | | | | | | | |
+| approx_count_distinct | approx_distinct | | S | | S | S | S | S | S | S | S | S | | S | | | | | | | | |
+| approx_percentile | | | | | | | | | | | | | | | | | | | | | | |
+| avg | avg | | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| bool_and | | | | | | | | | | | | | | | | | | | | | | |
+| bool_or | | | | | | | | | | | | | | | | | | | | | | |
+| collect_list | | | S | | | | | | | | | | | | | | | | | | | |
+| collect_set | | | S | | | | | | | | | | | | | | | | | | | |
+| corr | corr | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| count | count | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| count_if | count_if | | | | | S | S | S | S | S | | | | | | | | | | | | |
+| count_min_sketch | | | | | | | | | | | | | | | | | | | | | | |
+| covar_pop | covar_pop | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| covar_samp | covar_samp | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| every | | | | | | | | | | | | | | | | | | | | | | |
+| first | | first | S | | | | | | | | | | | | | | | | | | | |
+| first_value | | first_value | S | | | | | | | | | | | | | | | | | | | |
+| grouping | | | | | | | | | | | | | | | | | | | | | | |
+| grouping_id | | | | | | | | | | | | | | | | | | | | | | |
+| kurtosis | kurtosis | kurtosis | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| last | | last | S | | | | | | | | | | | | | | | | | | | |
+| last_value | | last_value | S | | | | | | | | | | | | | | | | | | | |
+| max | max | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| max_by | | | S | | | | | | | | | | | | | | | | | | | |
+| mean | avg | | S | ANSI OFF | | | | | | | | | | | | | | | | | | |
+| min | min | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| min_by | | | S | | | | | | | | | | | | | | | | | | | |
+| regr_avgx | regr_avgx | regr_avgx | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_avgy | regr_avgy | regr_avgy | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_count | regr_count | regr_count | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_r2 | regr_r2 | regr_r2 | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_intercept | regr_intercept | regr_intercept | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_slope | regr_slope | regr_slope | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_sxy | regr_sxy | regr_sxy | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_sxx | regr_sxx | regr_sxx | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| regr_syy | regr_syy | regr_syy | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| skewness | skewness | skewness | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| some | | | | | | | | | | | | | | | | | | | | | | |
+| std,stddev | stddev | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| stddev,std | stddev | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| stddev_pop | stddev_pop | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| stddev_samp | stddev_samp | | S | | | | S | S | S | S | S | | | | | | | | | | | |
+| sum | sum | | S | ANSI OFF | | S | S | S | S | S | | | | | | | | | | | | |
+| var_pop | var_pop | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| var_samp | var_samp | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| variance | variance | | S | | | S | S | S | S | S | | | | | | | | | | | | |
+| cume_dist | cume_dist | | S | | | | | | | | | | | | | | | | | | | |
+| dense_rank | dense_rank | | S | | | | | | | | | | | | | | | | | | | |
+| lag | | | S | | | | | | | | | | | | | | | | | | | |
+| lead | | | S | | | | | | | | | | | | | | | | | | | |
+| nth_value | nth_value | nth_value | PS | | | | | | | | | | | | | | | | | | | |
+| ntile | ntile | ntile | S | | | | | | | | | | | | | | | | | | | |
+| percent_rank | percent_rank | | S | | | | | | | | | | | | | | | | | | | |
+| rank | rank | | S | | | | | | | | | | | | | | | | | | | |
+| row_number | row_number | | S | | | | S | S | S | | | | | | | | | | | | | |
+| from_csv | | | | | | | | | | | | | | | | | | | | | | |
+| from_json | | | | | | | | | | | | | | | | | | | | | | |
+| get_json_object | json_extract_scalar | get_json_object | S | | | | | | | | | | | | | | | | | | S | |
+| json_array_length | json_array_length | | S | | | | | | | | | | | | | | | | | | S | |
+| json_tuple | | | | | | | | | | | | | | | | | | | | | | |
+| schema_of_csv | | | | | | | | | | | | | | | | | | | | | | |
+| schema_of_json | | | | | | | | | | | | | | | | | | | | | | |
+| to_csv | | | | | | | | | | | | | | | | | | | | | | |
+| to_json | | | | | | | | | | | | | | | | | | | | | | |
+| assert_true | | | S | | | | | | | | | | | | | | | | | | | |
+| coalesce | | | PS | | | | | | | | | | | | | | | | | | | |
+| crc32 | crc32 | | S | | | | | | | | | | | S | | | | | | | | |
+| current_user | | | S* | | | | | | | | | | | S | | | | | | | | |
+| current_catalog | | | S | | | | | | | | | | | | | | | | | | | |
+| current_database | | | S | | | | | | | | | | | | | | | | | | | |
+| greatest | greatest | greatest | S | | | | | | S | S | S | S | S | | | | | | | | | |
+| hash | hash | hash | S | | S | S | S | S | S | S | S | | | | | | | | | | | |
+| inline | | | | | | | | | | | | | | | | | | | | | | |
+| inline_outer | | | | | | | | | | | | | | | | | | | | | | |
+| input_file_name | | | S | | | | | | | | | | | | | | | | | | | |
+| input_file_block_length | | | S | | | | | | | | | | | | | | | | | | | |
+| input_file_block_start | | | S | | | | | | | | | | | | | | | | | | | |
+| java_method | | | | | | | | | | | | | | | | | | | | | | |
+| least | least | least | S | | | | | | S | S | S | S | S | | | | | | | | | |
+| md5 | md5 | | S | | | S | | | | | | | | | | | | | | | | |
+| monotonically_increasing_id | | | S | | | | | | | | | | | | | | | | | | | |
+| nanvl | | | S | | | | | | | | | | | | | | | | | | | |
+| nvl | | | | | | | | | | | | | | | | | | | | | | |
+| nvl2 | | | | | | | | | | | | | | | | | | | | | | |
+| raise_error | | raise_error | S | | | | | | | | | | | | | | | | | | | |
+| reflect | | | | | | | | | | | | | | | | | | | | | | |
+| sha | | | S | | | | | | | | | | | S | | | | | | | | |
+| sha1 | sha1 | sha1 | S | | | | | | | | | | | S | | | | | | | | |
+| sha2 | | sha2 | S | | | | | | | | | | | S | | | | | | | | |
+| spark_partition_id | | | S | | | | | | | | | | | | | | | | | | | |
+| stack | | | S | | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S | S |
+| xxhash64 | xxhash64 | xxhash64 | | | | | | | | | | | | | | | | | | | | |
+| uuid | uuid | uuid | S | | | | | | | | | | | | | | | | | | | |
+| rand | rand | rand | S | | | | | | | | | | | | | | | | | | | |
+| try_add | | | S | | | | | | | | | | | | | | | | | | | |
+| try_substract | | | S | | | | | | | | | | | | | | | | | | | |
+| try_multiply | | | S | | | | | | | | | | | | | | | | | | | |
+| try_divide | | | S | | | | | | | | | | | | | | | | | | | |
+
diff --git a/docs/velox-backend/velox-backend-troubleshooting.md b/docs/velox-backend/velox-backend-troubleshooting.md
new file mode 100644
index 0000000..440e0ce
--- /dev/null
+++ b/docs/velox-backend/velox-backend-troubleshooting.md
@@ -0,0 +1,100 @@
+---
+layout: page
+title: Troubleshooting
+nav_order: 3
+parent: Velox Backend
+grand_parent: Documentations
+permalink: /docs/velox-backend/troubleshooting/
+---
+## Troubleshooting
+
+### Fatal error after native exception is thrown
+
+We depend on checking exceptions thrown from native code to validate whether a spark plan can
+be really offloaded to native engine. But if `libunwind-dev` is installed, native exception will not be
+caught and will interrupt the program. So far, we observed this fatal error can happen only on Ubuntu 20.04.
+Please remove `libunwind-dev` and then re-build the project to address this issue.
+
+`sudo apt-get purge --auto-remove libunwind-dev`
+
+### Jar conflict issue
+
+With the latest version of Gluten, there should not be any jar conflict issue anymore. If you still get hit with
+such issue, please follow the below instructions.
+
+The potentially conflicting libraries include protobuf (Both Velox and CK backend), flatbuffers (Velox backend), and arrow-* (Velox backend).
+These libraries are compiled from source and packed into Gluten's jar. Jvm should search them from Gluten.jar firstly and load them. But for
+some reason jvm loads the jars from spark_home/jars which causes conflict. You may use below commands to remove the jars from spark_home/jars.
+We are still investigating the root cause. Welcome to share if you have good solution.
+
+```
+rm -rf $SPARK_HOME/jars/protobuf-*
+# velox backend only
+rm -rf $SPARK_HOME/jars/flatbuffers-*
+rm -rf $SPARK_HOME/jars/arrow-*
+```
+
+### Incompatible class error when using native writer
+Gluten native writer overwrite some vanilla spark classes. Therefore, when running a program that uses gluten, it is essential to ensure that
+the gluten jar is loaded prior to the vanilla spark jar. In this section, we will provide some configuration settings in
+`$SPARK_HOME/conf/spark-defaults.conf` for Yarn client, Yarn cluster, and Local&Standalone mode to guarantee that the gluten jar is prioritized.
+
+#### Configurations for Yarn Client mode
+
+```
+// spark will upload the gluten jar to hdfs and then the nodemanager will fetch the gluten jar before start the executor process. Here also can set the spark.jars.
+spark.files = {absolute_path}/gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+// The absolute path on running node
+spark.driver.extraClassPath={absolute_path}/gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+// The relative path under the executor working directory
+spark.executor.extraClassPath=./gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+```
+
+#### Configurations for Yarn Cluster mode
+```
+spark.driver.userClassPathFirst = true
+spark.executor.userClassPathFirst = true
+
+spark.files = {absolute_path}/gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+// The relative path under the executor working directory
+spark.driver.extraClassPath=./gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+// The relative path under the executor working directory
+spark.executor.extraClassPath=./gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+```
+#### Configurations for Local & Standalone mode
+```
+// The absolute path on running node
+spark.driver.extraClassPath={absolute_path}/gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+// The absolute path on running node
+spark.executor.extraClassPath={absolute_path}/gluten-<spark-version>-<gluten-version>-SNAPSHOT-jar-with-dependencies.jar
+```
+
+### Invalid pointer error
+
+If the below error is reported at runtime, please re-build gluten with `--compile_arrow_java=ON`, then redeploy Gluten jar.
+
+```
+*** Error in `/usr/local/jdk1.8.0_381/bin/java': free(): invalid pointer: 0x00007f36cb5cec80 ***
+======= Backtrace: =========
+/lib64/libc.so.6(+0x7d1fd)[0x7f38c29da1fd]
+/lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7f36cb3370d2]
+/lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7f36cb337523]
+/lib64/libstdc++.so.6(+0x71495)[0x7f36cb338495]
+/lib64/libpthread.so.0(pthread_once+0x50)[0x7f38c3147be0]
+/lib64/libstdc++.so.6(+0x714e1)[0x7f36cb3384e1]
+/lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7f36cb338523]
+/lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7f36cb33537c]
+/tmp/jnilib-645156599284574767.tmp(+0x2a90)[0x7f375d235a90]
+/lib64/ld-linux-x86-64.so.2(+0xf4e3)[0x7f38c33664e3]
+/lib64/ld-linux-x86-64.so.2(+0x13b04)[0x7f38c336ab04]
+/lib64/ld-linux-x86-64.so.2(+0xf2f4)[0x7f38c33662f4]
+/lib64/ld-linux-x86-64.so.2(+0x1321b)[0x7f38c336a21b]
+/lib64/libdl.so.2(+0x102b)[0x7f38c2d1f02b]
+/lib64/ld-linux-x86-64.so.2(+0xf2f4)[0x7f38c33662f4]
+/lib64/libdl.so.2(+0x162d)[0x7f38c2d1f62d]
+/lib64/libdl.so.2(dlopen+0x31)[0x7f38c2d1f0c1]
+/usr/local/jdk1.8.0_381/jre/lib/amd64/server/libjvm.so(+0x9292b1)[0x7f38c22732b1]
+/usr/local/jdk1.8.0_381/jre/lib/amd64/server/libjvm.so(JVM_LoadLibrary+0xa1)[0x7f38c205e0c1]
+/usr/local/jdk1.8.0_381/jre/lib/amd64/libjava.so(Java_java_lang_ClassLoader_00024NativeLibrary_load+0x1ac)
+...
+```
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org