You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by we...@apache.org on 2024/08/07 15:35:31 UTC
(incubator-gluten-site) branch main updated: Add Archives for v1.1.1 documentations (#19)
This is an automated email from the ASF dual-hosted git repository.
weitingchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten-site.git
The following commit(s) were added to refs/heads/main by this push:
new fad56c6 Add Archives for v1.1.1 documentations (#19)
fad56c6 is described below
commit fad56c6076692efdf3d2a83a253299b422019cf3
Author: Wei-Ting Chen <we...@apache.org>
AuthorDate: Wed Aug 7 23:35:27 2024 +0800
Add Archives for v1.1.1 documentations (#19)
---
archives/index.md | 10 +
archives/v1.1.1/developers/CppCodingStyle.md | 183 ++++
archives/v1.1.1/developers/HowTo.md | 157 ++++
archives/v1.1.1/developers/HowToRelease.md | 249 ++++++
archives/v1.1.1/developers/MicroBenchmarks.md | 308 +++++++
archives/v1.1.1/developers/NewToGluten.md | 378 +++++++++
.../v1.1.1/developers/SubstraitModifications.md | 37 +
archives/v1.1.1/developers/UsingGperftoolsInCH.md | 26 +
archives/v1.1.1/developers/UsingJemallocWithCH.md | 40 +
archives/v1.1.1/developers/docker_centos7.md | 54 ++
archives/v1.1.1/developers/docker_centos8.md | 51 ++
archives/v1.1.1/developers/docker_ubuntu22.04.md | 63 ++
archives/v1.1.1/developers/index.md | 12 +
.../developers/velox-function-development-guide.md | 48 ++
archives/v1.1.1/docs/ClickHouse.md | 10 +
archives/v1.1.1/docs/GettingStarted_ClickHouse.md | 702 ++++++++++++++++
archives/v1.1.1/docs/GettingStarted_Velox.md | 917 +++++++++++++++++++++
archives/v1.1.1/docs/Velox.md | 10 +
archives/v1.1.1/docs/VeloxBuildGuide.md | 78 ++
archives/v1.1.1/docs/VeloxGCS.md | 41 +
archives/v1.1.1/docs/VeloxS3.md | 75 ++
archives/v1.1.1/docs/index.md | 12 +
archives/v1.1.1/images/apache-incubator.svg | 1 +
archives/v1.1.1/images/gluten-logo-blue.png | Bin 0 -> 14469 bytes
archives/v1.1.1/index.md | 11 +
contact-us.md | 2 +-
contributing.md | 2 +-
release.md | 2 +-
28 files changed, 3476 insertions(+), 3 deletions(-)
diff --git a/archives/index.md b/archives/index.md
new file mode 100644
index 0000000..87d845d
--- /dev/null
+++ b/archives/index.md
@@ -0,0 +1,10 @@
+---
+layout: page
+title: Archives
+nav_order: 4
+has_children: true
+permalink: /archives/
+---
+# Gluten Documents by version
+
+
diff --git a/archives/v1.1.1/developers/CppCodingStyle.md b/archives/v1.1.1/developers/CppCodingStyle.md
new file mode 100644
index 0000000..afa7c8e
--- /dev/null
+++ b/archives/v1.1.1/developers/CppCodingStyle.md
@@ -0,0 +1,183 @@
+---
+layout: page
+title: v1.1.1/CPP Code Style
+nav_order: 4
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/cpp_code_style/
+---
+# Gluten CPP Core Guidelines
+
+This is a set of CPP core guidelines for Gluten. The aim is to make the codebase
+simpler, more efficient, more maintainable by promoting consistency and according
+to best practices.
+
+## Philosophy
+
+Philosophical rules are generally not measurable. However, they are valuable. For
+Gluten CPP coding, there are a few Philosophical rules as the following.
+
+* Write in ISO Standard C++.
+* Standard API first, the CPP programming APIs are priority to system calls.
+* Write code consistently. It's good for understanding and maintaining.
+* Keep simple, making code clear and easy to read.
+* Optimize code for reader, not for writer. Thus, more time will be spent reading
+ code than writing it.
+* Make it work, and then make it better or faster.
+* Don't import any complexity if possible. Collaborate with minimal knowledge
+ consensus.
+
+## Code Formatting
+
+Many aspects of C++ coding style will be covered by clang-format, such as spacing,
+line width, indentation and ordering (for includes, using directives and etc).
+
+* Always ensure your code is compatible with clang-format-12 for Velox backend.
+* `dev/formatcppcode.sh` is provided for formatting Velox CPP code.
+
+## Naming Conventions
+
+* Use **PascalCase** for types (class, struct, enum, type alias, type
+ template parameter) and file name.
+* Use **camelCase** for function, member and local variable, and non-type
+ template parameter.
+* Use **camelCase_** for private and protected member variable.
+* Use **snake_case** for namespace name and build target.
+* Use **UPPER_SNAKE_CASE** for macro.
+* Use **kPascalCase** for static constant and enumerator.
+
+## Designs
+
+* No over design.
+* No negation of negation, `isValid` is better than `isNotInvalid`.
+* Avoid corner case, and common case first.
+* Express ideas directly, don't let me think.
+* Make a check for the arguments in the interface between modules, and don't make a check
+ in the inner implementation, use `assert` in the private implementation instead of too
+ much safe check.
+
+## Source File & Header File
+
+* All header files must have a single-inclusion guard using `#pragma once`
+* Always use `.h` as header file suffix, not `.hpp`.
+* Always use `.cc` as source file suffix, neither `.cpp` nor `.cxx`.
+* One file should contain one main class, and the file name should be consistent with
+ the main class name.
+ * Obvious exception: files used for defining various misc functions.
+* If a header file has a corresponding source file, they should have the same file
+ name with different suffix, such as `a.h vs a.cc`.
+* If a function is declared in the file `a.h`, ensure it's defined in the corrosponding
+ source file `a.cc`, do not define it in other files.
+* No deep source directory for CPP files, not do it as JAVA.
+* Include header files should satisfy the following rules.
+ * Include the necessary header files, which means the source file (.cc) containing
+ the only one line `#include "test.h"` can be compiled successfully without
+ including any other header files.
+ * Do not include any unnecessary header files, the more including, the slower
+ compiling.
+ * In one word, no more, no less, just as needed.
+
+## Class
+
+* Base class name doesn't end with Base, use `Backend` instead of `BackendBase`.
+* Ensure one class does one thing, and follows the single responsibility principle.
+* No big class, No huge class, No too much interfaces.
+* Distinguish interface from implementation, make implementations private.
+* When designing a class hierarchy, distinguish between interface inheritance
+ and implementation inheritance.
+ * Ensure that public inheritance represent the relation of `is-a`.
+ * Ensure that private inheritance represent the relation of `implements-with`.
+* Don't make a function `virtual` without reason.
+* Ensure the polymorphic base class has a `virtual` deconstructor.
+* Use `override` to make overriding explicit and to make the compiler work.
+* Use `const` to mark the member function read-only as far as possible.
+* When you try to define a `copy constructor` or a `operator=` for a class, remember
+ the `Rule of three/five/zero`.
+
+## Function
+
+* Make functions short and simple.
+* Calling a meaningful function is more readable than writing too many statements
+ in place, but the performance-sensitive code path is an exception.
+* Give the function a good name, how to check whether the function name is good
+ or not.
+ * When you read it loudly, you feel smooth.
+ * The information can be represented by arguments should not be encoded into
+ the function name. such as. use `get(size_t index)` instead of `getByIndex`.
+* A function should focus on a single logic operation.
+* A function should do as the name meaning.
+ * do everything converd by the function name
+ * don't do anything not convered by the function name
+
+## Variable
+
+* Make variable names simple and meaningful.
+* Don't group all your variables at the top of the scope, it's an outdated habit.
+* Declare variables as close to the usage point as possible.
+
+## Constant
+
+* Prefer const variables to using preprocessor (`#define`) to define constant values.
+
+## Macro
+
+* Macros downgrade readability, break mind, and affect debug.
+* Macros have side effects.
+* Use macros cautiously and carefully.
+* Consider using `const` variables or `inline` functions to replace macros.
+* Consider defining macros with the wrap of `do {...} while (0)`
+* Avoid using 3rd party library macros directly.
+
+## Namespace
+
+* Don't `using namespace xxx` in header files. Instead, you can do this in source files.
+ But it's still not encouraged.
+* Place all Gluten CPP codes under `namespace gluten` because one level namespace
+ is enough. No nested namespace. Nested namespaces bring mess.
+* The anonymous namespace is recommended for defining file level classes, functions
+ and variables. It's used to place file scoped static functions and variables.
+
+## Resource Management
+
+* Use handles and RAII to manage resources automatically.
+* Immediately give the result of an explicit resource allocation to a manager
+ object.
+* Prefer scoped objects and stack objects.
+* Use raw pointers to denote individual objects.
+* Use `pointer + size_t` to denote array objects if you don't want to use containers.
+* A raw pointer (a `T*`) is non-owning.
+* A raw reference (a `T&`) is non-owning.
+* Understand the difference of `unique_ptr`, `shared_ptr`, `weak_ptr`.
+ * `unique_ptr` represents ownership, but not share ownership. `unique_ptr` is
+ equivalent to RAII, release the resource when the object is destructed.
+ * `shared_ptr` represents shared ownership by use-count. It is more expensive
+ that `unqiue_ptr`.
+ * `weak_ptr` models temporary ownership. It is useful in breaking reference cycles
+ formed by objects managed by `shared_ptr`.
+* Use `unique_ptr` or `shared_ptr` to represent ownership.
+* Prefer `unique_ptr` over `shared_ptr` unless you need to share ownership.
+* Use `make_unique` to make `unique_ptr`s.
+* Use `make_shared` to make `shared_ptr`s.
+* Take smart pointers as parameters only to explicitly express lifetime semantics.
+* **For general use**, take `T*` or `T&` arguments rather than smart pointers.
+
+## Exception
+
+* The exception specifications are changing always. The difference between various CPP
+ standards is big, so we should use exception cautiously in Gluten.
+* Prefer `return code` to throwing exceptions.
+* Prefer compile-time checking to run-time checking.
+* Encapsulate messy constructors, rather than spreading through the code.
+
+## Code Comment
+
+* Add necessary comments. The comment is not the more the better, also not the
+ less the better.
+* Good comment makes obscure code easily understood. It's unnecessary to add comments for
+ quite obvious code.
+
+## References
+
+* [CppCoreGuidelines](https://github.com/fluz/CppCoreGuidelines)
+* [Velox CODING_STYLE](https://github.com/facebookincubator/velox/blob/main/CODING_STYLE.md)
+* Thanks Gluten developers for their wise suggestions and helps.
diff --git a/archives/v1.1.1/developers/HowTo.md b/archives/v1.1.1/developers/HowTo.md
new file mode 100644
index 0000000..8911082
--- /dev/null
+++ b/archives/v1.1.1/developers/HowTo.md
@@ -0,0 +1,157 @@
+---
+layout: page
+title: v1.1.1/How To Use Gluten
+nav_order: 2
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/howto/
+---
+There are some common questions about developing, debugging and testing been asked again and again. In order to help the developers to contribute
+to Gluten as soon as possible, we collected these frequently asked questions, and organized them in the form of Q&A. It's convenient for the developers
+to check and learn.
+
+when you encountered a new problem and then resolved it, please add a new item to this document if you think it may be helpful to the other developers.
+
+We use `gluten_home` to represent the home directory of Gluten in this document.
+
+# How to understand the key work of Gluten?
+
+The Gluten worked as the role of bridge, it's a middle layer between the Spark and the native execution library.
+
+The Gluten is responsibility for validating whether the operators of the Spark plan can be executed by the native engine or not. If yes, the Gluten
+transforms Spark plan to Substrait plan, and then send the Substrait plan to the native engine.
+
+The Gluten codes consist of two parts: the C++ codes and the Java/Scala codes.
+1. All C++ codes are placed under the directory of `gluten_home/cpp`, the Java/Scala codes are placed under several directories, such as
+ `gluten_home/gluten-core` `gluten_home/gluten-data` `gluten_home/backends-velox`.
+2. The Java/Scala codes are responsibility for validating and transforming the execution plan. Source data should also be provided, the source data may
+ come from files or other forms such as networks.
+3. The C++ codes take the Substrait plan and the source data as inputs and transform the Substrait plan to the corresponding backend plan. If the backend
+ is Velox, the Substrait plan will be transformed to the Velox plan, and then be executed.
+
+JNI is a programming technology of invoking C++ from Java. All JNI interfaces are defined in the file `JniWrapper.cc` under the directory `jni`.
+
+# How to debug in Gluten?
+
+## 1 How to debug C++
+If you don't concern about the Scala/Java codes and just want to debug the C++ codes executed in native engine, you may debug the C++ via benchmarks
+with GDB.
+
+To debug C++, you have to generate the example files, the example files consist of:
+- A file contained Substrait plan in JSON format
+- One or more input data files in Parquet format
+
+You can generate the example files by the following steps:
+
+1. build Velox and Gluten CPP
+```
+gluten_home/dev/builddeps-veloxbe.sh --build_tests=ON --build_benchmarks=ON --build_type=Debug
+```
+- Compiling with `--build_type=Debug` is good for debugging.
+- The executable file `generic_benchmark` will be generated under the directory of `gluten_home/cpp/build/velox/benchmarks/`.
+
+2. build Gluten and generate the example files
+```
+cd gluten_home
+mvn clean package -Pspark-3.2 -Pbackends-velox -Prss
+mvn test -Pspark-3.2 -Pbackends-velox -Prss -pl backends-velox -am -DtagsToInclude="io.glutenproject.tags.GenerateExample" -Dtest=none -DfailIfNoTests=false -Darrow.version=11.0.0-gluten -Dexec.skip
+```
+- After the above operations, the examples files are generated under `gluten_home/backends-velox`
+- You can check it by the command `tree gluten_home/backends-velox/generated-native-benchmark/`
+- You may replace `-Pspark-3.2` with `-Pspark-3.3` if your spark's version is 3.3
+```shell
+$ tree gluten_home/backends-velox/generated-native-benchmark/
+gluten_home/backends-velox/generated-native-benchmark/
+├── example.json
+├── example_lineitem
+│ ├── part-00000-3ec19189-d20e-4240-85ae-88631d46b612-c000.snappy.parquet
+│ └── _SUCCESS
+└── example_orders
+ ├── part-00000-1e66fb98-4dd6-47a6-8679-8625dbc437ee-c000.snappy.parquet
+ └── _SUCCESS
+```
+
+3. now, run benchmarks with GDB
+```
+cd gluten_home/cpp/build/velox/benchmarks/
+gdb generic_benchmark
+```
+- When GDB load `generic_benchmark` successfully, you can set `breakpoint` on the `main` function with command `b main`, and then run with command `r`,
+ then the process `generic_benchmark` will start and stop at the `main` function.
+- You can check the variables' state with command `p variable_name`, or execute the program line by line with command `n`, or step-in the function been
+ called with command `s`.
+- Actually, you can debug `generic_benchmark` with any gdb commands as debugging normal C++ program, because the `generic_benchmark` is a pure C++
+ executable file in fact.
+
+4. `gdb-tui` is a valuable feature and is worth trying. You can get more help from the online docs.
+[gdb-tui](https://sourceware.org/gdb/onlinedocs/gdb/TUI.html)
+
+5. you can start `generic_benchmark` with specific JSON plan and input files
+- If you omit them, the `example.json, example_lineitem + example_orders` under the directory of `gluten_home/backends-velox/generated-native-benchmark`
+ will be used as default.
+- You can also edit the file `example.json` to custom the Substrait plan or specify the inputs files placed in the other directory.
+
+6. get more detail information about benchmarks from [MicroBenchmarks](./MicroBenchmarks.md)
+
+## 2 How to debug plan validation process
+Gluten will validate generated plan before execute it, and validation usually happens in native side, so we provide a utility to help debug validation process in native side.
+
+1. Run query with conf `spark.gluten.sql.debug=true`, and you will find generated plan be printed in stderr with json format, save it as `plan.json` for example.
+2. Compile cpp part with `--build_benchmarks=ON`, then check `plan_validator_util` executable file in `gluten_home/cpp/build/velox/benchmarks/`.
+3. Run or debug with `./plan_validator_util <path>/plan.json`
+
+## 3 How to debug Java/Scala
+wait to add
+
+## 4 How to debug with core-dump
+wait to complete
+```
+cd the_directory_of_core_file_generated
+gdb gluten_home/cpp/build/releases/libgluten.so 'core-Executor task l-2000883-1671542526'
+
+```
+- the `core-Executor task l-2000883-1671542526` represents the core file name.
+
+# How to run TPC-H on Velox backend
+
+Now, both Parquet and DWRF format files are supported, related scripts and files are under the directory of `gluten_home/backends-velox/workload/tpch`.
+The file `README.md` under `gluten_home/backends-velox/workload/tpch` offers some useful help but it's still not enough and exact.
+
+One way of run TPC-H test is to run velox-be by workflow, you can refer to [velox_be.yml](https://github.com/oap-project/gluten/blob/main/.github/workflows/velox_be.yml#L90)
+
+Here will explain how to run TPC-H on Velox backend with the Parquet file format.
+1. First step, prepare the datasets, you have two choices.
+ - One way, generate Parquet datasets using the script under `gluten_home/backends-velox/workload/tpch/gen_data/parquet_dataset`, You can get help from the above
+ mentioned `README.md`.
+ - The other way, using the small dataset under `gluten_home/backends-velox/src/test/resources/tpch-data-parquet-velox` directly, If you just want to make simple
+ TPC-H testing, this dataset is a good choice.
+2. Second step, run TPC-H on Velox backend testing.
+ - Modify `gluten_home/backends-velox/workload/tpch/run_tpch/tpch_parquet.scala`.
+ - set `var parquet_file_path` to correct directory. If using the small dataset directly in the step one, then modify it as below
+ ```
+ var parquet_file_path = "gluten_home/backends-velox/src/test/resources/tpch-data-parquet-velox"
+ ```
+ - set `var gluten_root` to correct directory. If `gluten_home` is the directory of `/home/gluten`, then modify it as below
+ ```
+ var gluten_root = "/home/gluten"
+ ```
+ - Modify `gluten_home/backends-velox/workload/tpch/run_tpch/tpch_parquet.sh`.
+ - Set `GLUTEN_JAR` correctly. Please refer to the section of [Build Gluten with Velox Backend](../get-started/Velox.md/#2-build-gluten-with-velox-backend)
+ - Set `SPARK_HOME` correctly.
+ - Set the memory configurations appropriately.
+ - Execute `tpch_parquet.sh` using the below command.
+ - `cd gluten_home/backends-velox/workload/tpch/run_tpch/`
+ - `./tpch_parquet.sh`
+
+# How to run TPC-DS
+wait to add
+
+# How to track the memory exhaust problem
+When your gluten spark jobs failed because of OOM, you can track the memory allocation's call stack by configuring `spark.gluten.backtrace.allocation = true`.
+The above configuration will use `BacktraceAllocationListener` wrapping from `SparkAllocationListener` to create `VeloxMemoryManager`.
+
+`BacktraceAllocationListener` will check every allocation, if a single allocation bytes exceeds a fixed value or the accumulative allocation bytes exceeds 1/2/3...G,
+the call stack of memory allocation will be outputted to standard output, you can check the backtrace and get some valuable information about tracking the memory exhaust issues.
+
+You can also adjust the policy to decide when to backtrace, such as the fixed value.
+
diff --git a/archives/v1.1.1/developers/HowToRelease.md b/archives/v1.1.1/developers/HowToRelease.md
new file mode 100644
index 0000000..9dfb3cc
--- /dev/null
+++ b/archives/v1.1.1/developers/HowToRelease.md
@@ -0,0 +1,249 @@
+---
+layout: page
+title: v1.1.1/How To Release
+nav_order: 10
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/howtorelease/
+---
+# How to Release
+
+This section outlines the steps for releasing Apache Gluten (incubating) according to the Apache release guidelines.
+All projects under the Apache umbrella must adhere to the [Apache Release Policy](https://www.apache.org/legal/release-policy.html). This guide is designed to assist you in comprehending the policy and navigating the process of releasing projects at Apache.
+
+
+## Release Process
+
+1. Prepare the release artifacts.
+
+2. Upload the release artifacts to the SVN repository.
+
+3. Verify the release artifacts.
+
+4. Initiate a release vote.
+
+5. Announce the results and the release.
+
+
+### Prepare the release artifacts.
+
+1. Create a branch from the target git repository.
+
+2. Tag a RC and draft the release notes.
+
+3. Build and Sign the release artifacts (including source archives, binaries, ...etc).
+
+4. Generate checksums for the artifacts.
+
+
+#### How to Sign the release artifacts.
+
+1. Create a GPG key
+
+2. Add the GPG key to the KEYS file
+
+3. Sign the release artifacts with the GPG key.
+
+```
+# create a GPG key, after executing this command, select the first one RSA 和 RSA
+$ gpg --full-generate-key
+
+# list the GPG keys
+$ gpg --keyid-format SHORT --list-keys
+
+# upload the GPG key to the key server, xxx is the GPG key id
+# eg: pub rsa4096/4C21E346 2024-05-06 [SC], 4C21E346 is the GPG key id;
+$ gpg --keyserver keyserver.ubuntu.com --send-key xxx
+
+# append the GPG key to the KEYS file the svn repository
+# [IMPORTANT] Don't replace the KEYS file, just append the GPG key to the KEYS file.
+$ svn co https://dist.apache.org/repos/dist/release/incubator/answer/
+$ (gpg --list-sigs xxx@apache.org && gpg --export --armor xxx@apache.org) >> KEYS
+$ svn ci -m "add gpg key"
+
+# sign the release artifacts, xxxx is xxx@apache.org
+$ for i in *.tar.gz; do echo $i; gpg --local-user xxxx --armor --output $i.asc --detach-sig $i ; done
+```
+
+
+#### How to Generate checksums for the release artifacts.
+
+```
+# create the checksums
+$ for i in *.tar.gz; do echo $i; sha512sum $i > $i.sha512 ; done
+```
+
+
+### Upload the release artifacts to the SVN repository.
+
+1. Create a project directory in the SVN repository (1st time only).
+`https://dist.apache.org/repos/dist/dev/incubator/gluten/`
+
+2. Create a directory for the release artifacts in the SVN repository.
+`https://dist.apache.org/repos/dist/dev/incubator/gluten/{release-version}`
+release-version format: apache-gluten-#.#.#-rc#
+
+3. Upload the release artifacts to the SVN repository.
+```
+$ svn co https://dist.apache.org/repos/dist/dev/incubator/gluten/
+$ cp /path/to/release/artifacts/* ./{release-version}/
+$ svn add ./{release-version}/*
+$ svn commit -m "add Apache Answer release artifacts for {release-version}"
+```
+
+4. After the upload, please visit the link `https://dist.apache.org/repos/dist/dev/incubator/gluten/{release-version}` to verify if the file upload is successful or not.
+The upload release artifacts should be include
+```
+* apache-gluten-#.#.#-incubating-src.tar.gz
+* apache-gluten-#.#.#-incubating-src.tar.gz.asc
+* apache-gluten-#.#.#-incubating-src.tar.gz.sha512
+```
+
+
+### Verify the release artifacts.
+
+Please follow below steps to verify the release artifacts.
+
+1. Check if the Download links are valid.
+
+2. Check if the checksums and GPG signatures are valid.
+
+3. Check if the release artifacts name is qualified and match with the current release.
+
+4. Check if LICENSE and NOTICE files are correct.
+
+5. Check if the License Headers are included in all files if necessary.
+
+6. No unlicensed compiled archives bundled in source archive.
+
+
+#### How to Verify the Signatures
+
+Please follow below steps to verify the signatures.
+
+```
+# download KEYS
+$ curl https://dist.apache.org/repos/dist/release/incubator/gluten/KEYS > KEYS
+
+# import KEYS and trust the key, please replace the email address with the one you want to trust.
+$ gpg --import KEYS
+$ gpg --edit-key xxx@apache.org
+gpg> trust
+gpg> 5
+gpg> y
+gpg> quit
+
+# enter the directory where the release artifacts are located
+$ cd /path/to/release/artifacts
+
+# verify the signature
+$ for i in *.tar.gz; do echo $i; gpg --verify $i.asc $i ; done
+
+# if you see 'Good signature' in the output, it means the signature is valid.
+```
+
+
+#### How to Verify the checksums
+
+Please follow below steps to verify the checksums
+```
+# verify the checksums
+$ for i in *.tar.gz; do echo $i; sha512sum --check $i.sha512; done
+```
+
+
+### Initiate a release vote.
+
+1. Email a vote request to dev@gluten.apache.org, requiring at least 3 PPMC +1s.
+
+2. Allow 72 hours or until enough votes are collected.
+
+3. Share the vote outcome on the dev list.
+
+4. If successful, request a vote on general@incubator.apache.org, needing 3 PMC +1s.
+
+5. Wait 72 hours or for sufficient votes.
+
+6. Announce the results on the general list.
+
+Vote Email Template
+```
+[VOTE] Release Apache Gluten (Incubating) {release-version}
+
+Hello,
+
+ This is a call for vote to release Apache Gluten (Incubating) version {release-version}.
+
+ The vote thread:
+ https://lists.apache.org/thread/{id}
+
+ Vote Result:
+ https://lists.apache.org/thread/{id}
+
+ The release candidates:
+ https://dist.apache.org/repos/dist/dev/incubator/gluten/{release-version}/
+
+ Release notes:
+ https://github.com/apache/incubator-gluten/releases/tag/{release-version}
+
+ Git tag for the release:
+ https://github.com/apache/incubator-gluten/releases/tag/{release-version}
+
+ Git commit id for the release:
+ https://github.com/apache/incubator-gluten/commit/{id}
+
+ Keys to verify the Release Candidate:
+ https://downloads.apache.org/incubator/gluten/KEYS
+
+ The vote will be open for at least 72 hours or until the necessary number of votes are reached.
+
+ Please vote accordingly:
+
+ [ ] +1 approve
+ [ ] +0 no opinion
+ [ ] -1 disapprove with the reason
+
+ Checklist for reference:
+
+ [ ] Download links are valid.
+ [ ] Checksums and PGP signatures are valid.
+ [ ] Source code distributions have correct names matching the current release.
+ [ ] LICENSE and NOTICE files are correct for each Apache Answer repo.
+ [ ] All files have license headers if necessary.
+ [ ] No unlicensed compiled archives bundled in source archive.
+
+ To compile from the source, please refer to:
+
+ https://github.com/apache/incubator-gluten#building-from-source
+
+Thanks,
+<YOUR NAME>
+```
+
+
+### Announce the results and the release.
+
+
+Announce Email Template
+```
+Hello everyone,
+
+The Apache Gluten (Incubating) {release-version} has been released!
+
+Apache Gluten is a Q&A platform software for teams at any scale.
+Whether it's a community forum, help center, or knowledge management platform, you can always count on Apache Answer.
+
+Download Links: https://downloads.apache.org/incubator/gluten/
+
+Release Notes: https://github.com/apache/incubator-gluten/releases/tag/{release-version}
+
+Website: https://gluten.apache.org/
+
+Resources:
+- Issue: https://github.com/apache/incubator-gluten/issues
+- Mailing list: dev@gluten.apache.org
+
+Thanks,
+<YOUR NAME>
+
+```
diff --git a/archives/v1.1.1/developers/MicroBenchmarks.md b/archives/v1.1.1/developers/MicroBenchmarks.md
new file mode 100644
index 0000000..da24278
--- /dev/null
+++ b/archives/v1.1.1/developers/MicroBenchmarks.md
@@ -0,0 +1,308 @@
+---
+layout: page
+title: v1.1.1/Micro Benchmarks for Velox Backend
+nav_order: 3
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/microbenchmark/
+---
+
+# Generate Micro Benchmarks for Velox Backend
+
+**This document explains how to use the existing micro benchmark template in Gluten Cpp.**
+
+A micro benchmark for Velox backend is provided in Gluten Cpp to simulate the execution of a first or middle stage in
+Spark.
+It serves as a more convenient alternative to debug in Gluten Cpp comparing with directly debugging in a Spark job.
+Developers can use it to create their own workloads, debug in native process, profile the hotspot and do optimizations.
+
+To simulate a first stage, you need to dump the Substrait plan and input split info into two JSON files. The input URIs
+of the splits should be exising file locations, which can be either local or HDFS paths.
+
+To simulate a middle stage, in addition to the JSON file, you also need to save the input data of this stage into
+Parquet files.
+The benchmark will load the data into Arrow format, then add Arrow2Velox to feed
+the data into Velox pipeline to reproduce the reducer stage. Shuffle exchange is not included.
+
+Please refer to the sections below to learn how to dump the Substrait plan and create the input data files.
+
+## Try the example
+
+To run a micro benchmark, user should provide one file that contains the Substrait plan in JSON format, and optional
+one or more input data files in parquet format.
+The commands below help to generate example input files:
+
+```shell
+cd /path/to/gluten/
+./dev/buildbundle-veloxbe.sh --build_tests=ON --build_benchmarks=ON
+
+# Run test to generate input data files. If you are using spark 3.3, replace -Pspark-3.2 with -Pspark-3.3
+mvn test -Pspark-3.2 -Pbackends-velox -Prss -pl backends-velox -am \
+-DtagsToInclude="io.glutenproject.tags.GenerateExample" -Dtest=none -DfailIfNoTests=false -Darrow.version=11.0.0-gluten -Dexec.skip
+```
+
+The generated example files are placed in gluten/backends-velox:
+
+```shell
+$ tree gluten/backends-velox/generated-native-benchmark/
+gluten/backends-velox/generated-native-benchmark/
+├── example.json
+├── example_lineitem
+│ ├── part-00000-3ec19189-d20e-4240-85ae-88631d46b612-c000.snappy.parquet
+│ └── _SUCCESS
+└── example_orders
+ ├── part-00000-1e66fb98-4dd6-47a6-8679-8625dbc437ee-c000.snappy.parquet
+ └── _SUCCESS
+```
+
+Run micro benchmark with the generated files as input. You need to specify the **absolute** path to the input files:
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /home/sparkuser/github/oap-project/gluten/backends-velox/generated-native-benchmark/example.json \
+--data /home/sparkuser/github/oap-project/gluten/backends-velox/generated-native-benchmark/example_orders/part-00000-1e66fb98-4dd6-47a6-8679-8625dbc437ee-c000.snappy.parquet,\
+/home/sparkuser/github/oap-project/gluten/backends-velox/generated-native-benchmark/example_lineitem/part-00000-3ec19189-d20e-4240-85ae-88631d46b612-c000.snappy.parquet \
+--threads 1 --iterations 1 --noprint-result --benchmark_filter=InputFromBatchStream
+```
+
+The output should be like:
+
+```shell
+2022-11-18T16:49:56+08:00
+Running ./generic_benchmark
+Run on (192 X 3800 MHz CPU s)
+CPU Caches:
+ L1 Data 48 KiB (x96)
+ L1 Instruction 32 KiB (x96)
+ L2 Unified 2048 KiB (x96)
+ L3 Unified 99840 KiB (x2)
+Load Average: 0.28, 1.17, 1.59
+***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
+-- Project[expressions: (n3_0:BIGINT, ROW["n1_0"]), (n3_1:VARCHAR, ROW["n1_1"])] -> n3_0:BIGINT, n3_1:VARCHAR
+ Output: 535 rows (65.81KB, 1 batches), Cpu time: 36.33us, Blocked wall time: 0ns, Peak memory: 1.00MB, Memory allocations: 3, Threads: 1
+ queuedWallNanos sum: 2.00us, count: 2, min: 0ns, max: 2.00us
+ -- HashJoin[RIGHT SEMI (FILTER) n0_0=n1_0] -> n1_0:BIGINT, n1_1:VARCHAR
+ Output: 535 rows (65.81KB, 1 batches), Cpu time: 191.56us, Blocked wall time: 0ns, Peak memory: 2.00MB, Memory allocations: 8
+ HashBuild: Input: 582 rows (16.45KB, 1 batches), Output: 0 rows (0B, 0 batches), Cpu time: 1.84us, Blocked wall time: 0ns, Peak memory: 1.00MB, Memory allocations: 3, Threads: 1
+ distinctKey0 sum: 583, count: 1, min: 583, max: 583
+ queuedWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ rangeKey0 sum: 59748, count: 1, min: 59748, max: 59748
+ HashProbe: Input: 37897 rows (296.07KB, 1 batches), Output: 535 rows (65.81KB, 1 batches), Cpu time: 189.71us, Blocked wall time: 0ns, Peak memory: 1.00MB, Memory allocations: 5, Threads: 1
+ queuedWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ -- ArrowStream[] -> n0_0:BIGINT
+ Input: 0 rows (0B, 0 batches), Output: 37897 rows (296.07KB, 1 batches), Cpu time: 1.29ms, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+ -- ArrowStream[] -> n1_0:BIGINT, n1_1:VARCHAR
+ Input: 0 rows (0B, 0 batches), Output: 582 rows (16.45KB, 1 batches), Cpu time: 894.22us, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+
+-----------------------------------------------------------------------------------------------------------------------------
+Benchmark Time CPU Iterations UserCounters...
+-----------------------------------------------------------------------------------------------------------------------------
+InputFromBatchVector/iterations:1/process_time/real_time/threads:1 41304520 ns 23740340 ns 1 collect_batch_time=34.7812M elapsed_time=41.3113M
+
+```
+
+## Generate Substrait plan and input for any query
+
+First, build Gluten with `--build_benchmarks=ON`.
+
+```shell
+cd /path/to/gluten/
+./dev/buildbundle-veloxbe.sh --build_benchmarks=ON
+
+# For debugging purpose, rebuild Gluten with build type `Debug`.
+./dev/buildbundle-veloxbe.sh --build_benchmarks=ON --build_type=Debug
+```
+
+First, get the Stage Id from spark UI for the stage you want to simulate.
+And then re-run the query with below configurations to dump the inputs to micro benchmark.
+
+| Parameters | Description | Recommend Setting |
+|---------------------------------------------|----------------------------------------------------------------------------------------------------------------|-------------------|
+| spark.gluten.sql.benchmark_task.stageId | Spark task stage id | target stage id |
+| spark.gluten.sql.benchmark_task.partitionId | Spark task partition id, default value -1 means all the partition of this stage | 0 |
+| spark.gluten.sql.benchmark_task.taskId | If not specify partition id, use spark task attempt id, default value -1 means all the partition of this stage | target task attemp id |
+| spark.gluten.saveDir | Directory to save the inputs to micro benchmark, should exist and be empty. | /path/to/saveDir |
+
+
+Check the files in `spark.gluten.saveDir`. If the simulated stage is a first stage, you will get 3 types of dumped file:
+
+- Configuration file: INI formatted, file name `conf_[stageId]_[partitionId].ini`. Contains the configurations to init Velox backend and runtime session.
+- Plan file: JSON formatted, file name `plan_[stageId]_[partitionId].json`. Contains the substrait plan to the stage, without input file splits.
+- Split file: JSON formatted, file name `split_[stageId]_[partitionId]_[splitIndex].json`. There can be more than one split file in a first stage task. Contains the substrait plan piece to the input file splits.
+
+Run benchmark. By default, the result will be printed to stdout. You can use `--noprint-result` to suppress this output.
+
+Sample command:
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--conf /absolute_path/to/conf_[stageId]_[partitionId].ini \
+--plan /absolute_path/to/plan_[stageId]_[partitionId].json \
+--split /absolut_path/to/split_[stageId]_[partitionId]_0.parquet,/absolut_path/to/split_[stageId]_[partitionId]_1.parquet \
+--threads 1 --noprint-result
+```
+
+If the simulated stage is a middle stage, you will get 3 types of dumped file:
+
+- Configuration file: INI formatted, file name `conf_[stageId]_[partitionId].ini`. Contains the configurations to init Velox backend and runtime session.
+- Plan file: JSON formatted, file name `plan_[stageId]_[partitionId].json`. Contains the substrait plan to the stage.
+- Data file: Parquet formatted, file name `data_[stageId]_[partitionId]_[iteratorIndex].json`. There can be more than one input data file in a middle stage task. The input data files of a middle stage will be loaded as iterators to serve as the inputs for the pipeline:
+
+```json
+"localFiles": {
+"items": [
+{
+"uriFile": "iterator:0"
+}
+]
+}
+```
+
+Sample command:
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--conf /absolute_path/to/conf_[stageId]_[partitionId].ini \
+--plan /absolute_path/to/plan_[stageId]_[partitionId].json \
+--data /absolut_path/to/data_[stageId]_[partitionId]_0.parquet,/absolut_path/to/data_[stageId]_[partitionId]_1.parquet \
+--threads 1 --noprint-result
+```
+
+For some complex queries, stageId may cannot represent the Substrait plan input, please get the taskId from spark UI,
+and get your target parquet from saveDir.
+
+In this example, only one partition input with partition id 2, taskId is 36, iterator length is 2.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/complex_plan.json \
+--data /absolute_path/to/data_36_2_0.parquet,/absolute_path/to/data_36_2_1.parquet \
+--threads 1 --noprint-result
+```
+
+## Save ouput to parquet to analyze
+
+You can save the output to a parquet file to analyze.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/plan.json \
+--data /absolute_path/to/data.parquet
+--threads 1 --noprint-result --write-file=/absolute_path/to/result.parquet
+```
+
+## Add shuffle write process
+
+You can add the shuffle write process at the end of this stage. Note that this will ignore the `--write-file` option.
+
+```shell
+cd /path/to/gluten/cpp/build/velox/benchmarks
+./generic_benchmark \
+--plan /absolute_path/to/plan.json \
+--split /absolute_path/to/split.json \
+--threads 1 --noprint-result --with-shuffle
+```
+
+By default, the compression codec for shuffle outputs is LZ4. You can switch to other codecs by adding one of the
+following argument flags to the command:
+
+- --zstd: ZSTD codec, compression level 1
+- --qat-gzip: QAT GZIP codec, compression level 1
+- --qat-zstd: QAT ZSTD codec, compression level 1
+- --iaa-gzip: IAA GZIP codec, compression level 1
+
+Note using QAT or IAA codec requires Gluten cpp is built with these features.
+Please check the corresponding section in [Velox document](../get-started/Velox.md) first for how to setup, build and
+enable these features in Gluten.
+For QAT support, please
+check [Intel® QuickAssist Technology (QAT) support](../get-started/Velox.md#intel-quickassist-technology-qat-support).
+For IAA support, please
+check [Intel® In-memory Analytics Accelerator (IAA/IAX) support](../get-started/Velox.md#intel-in-memory-analytics-accelerator-iaaiax-support)
+
+## Simulate Spark with multiple processes and threads
+
+You can use below command to launch several processes and threads to simulate parallel execution on Spark. Each thread
+in the same process will be pinned to the core number starting from `--cpu`.
+
+Suppose running on a baremetal machine with 48C, 2-socket, HT-on, launching below command will utilize all vcores.
+
+```shell
+processes=24 # Same value of spark.executor.instances
+threads=8 # Same value of spark.executor.cores
+
+for ((i=0; i<${processes}; i++)); do
+ ./generic_benchmark --plan /path/to/plan.json --split /path/to/split.json --noprint-result --threads $threads --cpu $((i*threads)) &
+done
+```
+
+If you want to add the shuffle write process, you can specify multiple direcotries by setting environment
+variable `GLUTEN_SPARK_LOCAL_DIRS` to a comma-separated string for shuffle write to spread the I/O pressure to multiple
+disks.
+
+```shell
+mkdir -p {/data1,/data2,/data3}/tmp # Make sure each directory has been already created.
+export GLUTEN_SPARK_LOCAL_DIRS=/data1/tmp,/data2/tmp,/data3/tmp
+
+processes=24 # Same value of spark.executor.instances
+threads=8 # Same value of spark.executor.cores
+
+for ((i=0; i<${processes}; i++)); do
+ ./generic_benchmark --plan /path/to/plan.json --split /path/to/split.json --noprint-result --with-shuffle --threads $threads --cpu $((i*threads)) &
+done
+```
+
+### Run Examples
+
+We also provide some example inputs in [cpp/velox/benchmarks/data](../../cpp/velox/benchmarks/data).
+E.g. [generic_q5/q5_first_stage_0.json](../../cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0.json) simulates a
+first-stage in TPCH Q5, which has the the most heaviest table scan. You can follow below steps to run this example.
+
+1. Open [generic_q5/q5_first_stage_0.json](../../cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json) with
+ file editor. Search for `"uriFile": "LINEITEM"` and replace `LINEITEM` with the URI to one partition file in
+ lineitem. In the next line, replace the number in `"length": "..."` with the actual file length. Suppose you are
+ using the provided small TPCH table
+ in [cpp/velox/benchmarks/data/tpch_sf10m](../../cpp/velox/benchmarks/data/tpch_sf10m), the replaced JSON should be
+ like:
+
+```
+{
+ "items": [
+ {
+ "uriFile": "file:///path/to/gluten/cpp/velox/benchmarks/data/tpch_sf10m/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet",
+ "length": "1863237",
+ "parquet": {}
+ }
+ ]
+}
+```
+
+2. Launch multiple processes and multiple threads. Set `GLUTEN_SPARK_LOCAL_DIRS` and add --with-shuffle to the command.
+
+```
+mkdir -p {/data1,/data2,/data3}/tmp # Make sure each directory has been already created.
+export GLUTEN_SPARK_LOCAL_DIRS=/data1/tmp,/data2/tmp,/data3/tmp
+
+processes=24 # Same value of spark.executor.instances
+threads=8 # Same value of spark.executor.cores
+
+for ((i=0; i<${processes}; i++)); do
+ ./generic_benchmark --plan /path/to/gluten/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0.json --split /path/to/gluten/cpp/velox/benchmarks/data/generic_q5/q5_first_stage_0_split.json --noprint-result --with-shuffle --threads $threads --cpu $((i*threads)) &
+done >stdout.log 2>stderr.log
+```
+
+You can find the "elapsed_time" and other metrics in stdout.log. In below output, the "elapsed_time" is ~10.75s. If you
+run TPCH Q5 with Gluten on Spark, a single task in the same Spark stage should take about the same time.
+
+```
+------------------------------------------------------------------------------------------------------------------
+Benchmark Time CPU Iterations UserCounters...
+------------------------------------------------------------------------------------------------------------------
+SkipInput/iterations:1/process_time/real_time/threads:8 1317255379 ns 10061941861 ns 8 collect_batch_time=0 elapsed_time=10.7563G shuffle_compress_time=4.19964G shuffle_spill_time=0 shuffle_split_time=0 shuffle_write_time=1.91651G
+```
+
+
diff --git a/archives/v1.1.1/developers/NewToGluten.md b/archives/v1.1.1/developers/NewToGluten.md
new file mode 100644
index 0000000..3861bb5
--- /dev/null
+++ b/archives/v1.1.1/developers/NewToGluten.md
@@ -0,0 +1,378 @@
+---
+layout: page
+title: v1.1.1/New To Gluten
+nav_order: 1
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/newto/
+---
+Help users to debug and test with gluten.
+
+For intel internal developer, you could refer to internal wiki [New Employee Guide](https://wiki.ith.intel.com/display/HPDA/New+Employee+Guide) to get more information such as proxy settings,
+Gluten has cpp code and java/scala code, we can use some useful IDE to read and debug.
+
+# Environment
+
+Now gluten supports Ubuntu20.04, Ubuntu22.04, centos8, centos7 and macOS.
+
+## Openjdk8
+
+### Environment setting
+
+For root user, the environment variables file is `/etc/profile`, it will make effect for all the users.
+
+For other user, you can set in `~/.bashrc`.
+
+### Guide for ubuntu
+The default JDK version in ubuntu is java11, we need to set to java8.
+
+```bash
+apt install openjdk-8-jdk
+update-alternatives --config java
+java -version
+```
+
+`--config java` to config java executable path, `javac` and other commands can also use this command to config.
+For some other uses, we suggest to set `JAVA_HOME`.
+
+```bash
+export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
+JRE_HOME=$JAVA_HOME/jre
+export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
+# pay attention to $PATH double quote
+export PATH="$PATH:$JAVA_HOME/bin"
+```
+
+> Must set PATH with double quote in ubuntu.
+
+## Maven 3.6.3 or above
+
+[Maven Dowload Page](https://maven.apache.org/docs/history.html)
+And then set the environment setting.
+
+## GCC 9.4 or above
+
+# Compile gluten using debug mode
+
+If you want to just debug java/scala code, there is no need to compile cpp code with debug mode.
+You can just refer to [build-gluten-with-velox-backend](../get-started/Velox.md#2-build-gluten-with-velox-backend).
+
+If you need to debug cpp code, please compile the backend code and gluten cpp code with debug mode.
+
+```bash
+## compile velox backend with benchmark and tests to debug
+gluten_home/dev/builddeps-veloxbe.sh --build_tests=ON --build_benchmarks=ON --build_type=Debug
+```
+
+If you need to debug the tests in <gluten>/gluten-ut, You need to compile java code with `-P spark-ut`.
+
+# Java/scala code development with Intellij
+
+## Linux intellij local debug
+
+Install the linux intellij version, and debug code locally.
+
+- Ask your linux maintainer to install the desktop, and then restart the server.
+- If you use Moba-XTerm to connect linux server, you don't need to install x11 server, If not (e.g. putty), please follow this guide:
+[X11 Forwarding: Setup Instructions for Linux and Mac](https://www.businessnewsdaily.com/11035-how-to-use-x11-forwarding.html)
+
+- Download [intellij linux community version](https://www.jetbrains.com/idea/download/?fromIDE=#section=linux) to linux server
+- Start Idea, `bash <idea_dir>/idea.sh`
+
+Notes: Sometimes, your desktop may stop accidently, left idea running.
+
+```bash
+root@xx2:~bash idea-IC-221.5787.30/bin/idea.sh
+Already running
+root@xx2:~ps ux | grep intellij
+root@xx2:kill -9 <pid>
+```
+
+And then restart idea.
+
+## Windows/Mac intellij remote debug
+
+If you have Ultimate intellij, you can try to debug remotely.
+
+## Set up gluten project
+
+- Make sure you have compiled gluten.
+- Load the gluten by File->Open, select <gluten_home/pom.xml>.
+- Activate your profiles such as <backends-velox>, and Reload Maven Project, you will find all your need modules have been activated.
+- Create breakpoint and debug as you wish, maybe you can try `CTRL+N` to find `TestOperator` to start your test.
+
+## Java/Scala code style
+
+Intellij IDE supports importing settings for Java/Scala code style. You can import [intellij-codestyle.xml](../../dev/intellij-codestyle.xml) to your IDE.
+See [Intellij guide](https://www.jetbrains.com/help/idea/configuring-code-style.html#import-code-style).
+
+To generate a fix for Java/Scala code style, you can run one or more of the below commands according to the code modules involved in your PR.
+
+For Velox backend:
+```
+mvn spotless:apply -Pbackends-velox -Prss -Pspark-3.2 -Pspark-ut -DskipTests
+mvn spotless:apply -Pbackends-velox -Prss -Pspark-3.3 -Pspark-ut -DskipTests
+```
+For Clickhouse backend:
+```
+mvn spotless:apply -Pbackends-clickhouse -Pspark-3.2 -Pspark-ut -DskipTests
+mvn spotless:apply -Pbackends-clickhouse -Pspark-3.3 -Pspark-ut -DskipTests
+```
+
+# CPP code development with Visual Studio Code
+
+This guide is for remote debug. We will connect the remote linux server by `SSH`.
+Download the [windows vscode software](https://code.visualstudio.com/Download)
+The important leftside bar is:
+- Explorer (Project structure)
+- Search
+- Run and Debug
+- Extensions (Install C/C++ Extension Pack, Remote Development, GitLens at least, C++ Test Mate is also suggested)
+- Remote Explorer (Connect linux server by ssh command, click `+`, then input `ssh user@10.1.7.003`)
+- Manage (Settings)
+
+Input your password in the above pop-up window, it will take a few minutes to install linux vscode server in remote machine folder `~/.vscode-server`
+If download failed, delete this folder and try again.
+
+## Usage
+
+### Set up project
+
+File->Open Folder // select gluten folder
+Select cpp/CmakeList.txt as command prompt
+Select gcc version as command prompt
+
+### Settings
+
+VSCode support 2 ways to set user setting.
+
+- Manage->Command Palette(Open `settings.json`, search by `Preferences: Open Settings (JSON)`)
+- Manage->Settings (Common setting)
+
+### Build by vscode
+
+VSCode will try to compile the debug version in <gluten_home>/build.
+And we need to compile velox debug mode before, if you have compiled velox release mode, you just need to do.
+
+```bash
+# Build the velox debug version in <velox_home>/_build/debug
+make debug EXTRA_CMAKE_FLAGS="-DVELOX_ENABLE_PARQUET=ON -DENABLE_HDFS=ON -DVELOX_BUILD_TESTING=OFF -DVELOX_ENABLE_DUCKDB=ON -DVELOX_BUILD_TEST_UTILS=ON"
+```
+
+Then gluten will link velox debug library.
+Just click `build` in bottom bar, you will get intellisense search and link.
+
+### Debug
+
+The default compile command does not enable test and benchmark, so we cannot get any executable file
+Open the file in `<gluten_home>/.vscode/settings.json` (create if not exists)
+
+```json
+{
+ "cmake.configureArgs": [
+ "-DBUILD_BENCHMARKS=ON",
+ "-DBUILD_TESTS=ON"
+ ],
+ "C_Cpp.default.configurationProvider": "ms-vscode.cmake-tools"
+}
+```
+
+Then we can get some executables, take `velox_shuffle_writer_test` as example
+
+Click `Run and Debug` to create launch.json in `<gluten_home>/.vscode/launch.json`
+Click `Add Configuration` in the top of launch.json, select gdb launch or attach to exists program
+launch.json example
+
+```json
+{
+ // Use IntelliSense to learn about possible attributes.
+ // Hover to view descriptions of existing attributes.
+ // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+ "version": "0.2.0",
+ "configurations": [
+ {
+ "name": "velox shuffle writer test",
+ "type": "cppdbg",
+ "request": "launch",
+ "program": "/mnt/DP_disk1/code/gluten/cpp/build/velox/tests/velox_shuffle_writer_test",
+ "args": ["--gtest_filter=*TestSinglePartPartitioner*"],
+ "stopAtEntry": false,
+ "cwd": "${fileDirname}",
+ "environment": [],
+ "externalConsole": false,
+ "MIMode": "gdb",
+ "setupCommands": [
+ {
+ "description": "Enable pretty-printing for gdb",
+ "text": "-enable-pretty-printing",
+ "ignoreFailures": true
+ },
+ {
+ "description": "Set Disassembly Flavor to Intel",
+ "text": "-gdb-set disassembly-flavor intel",
+ "ignoreFailures": true
+ }
+ ]
+ },
+ {
+ "name": "benchmark test",
+ "type": "cppdbg",
+ "request": "launch",
+ "program": "/mnt/DP_disk1/code/gluten/cpp/velox/benchmarks/./generic_benchmark",
+ "args": ["/mnt/DP_disk1/code/gluten/cpp/velox/benchmarks/query.json", "--threads=1"],
+ "stopAtEntry": false,
+ "cwd": "${fileDirname}",
+ "environment": [],
+ "externalConsole": false,
+ "MIMode": "gdb",
+ "setupCommands": [
+ {
+ "description": "Enable pretty-printing for gdb",
+ "text": "-enable-pretty-printing",
+ "ignoreFailures": true
+ },
+ {
+ "description": "Set Disassembly Flavor to Intel",
+ "text": "-gdb-set disassembly-flavor intel",
+ "ignoreFailures": true
+ }
+ ]
+ }
+
+ ]
+}
+```
+
+> Change `name`, `program`, `args` to yours
+
+Then you can create breakpoint and debug in `Run and Debug` section.
+
+### Velox debug
+
+For some velox tests such as `ParquetReaderTest`, tests need to read the parquet file in `<velox_home>/velox/dwio/parquet/tests/examples`, you should let the screen on `ParquetReaderTest.cpp`, then click `Start Debuging`, otherwise you will raise No such file or directory exception
+
+## Usefule notes
+
+### Upgrade vscode
+
+No need to upgrade vscode version, if upgraded, will download linux server again, switch update mode to off
+Search `update` in Manage->Settings to turn off update mode
+
+### Colour setting
+
+```json
+"workbench.colorTheme": "Quiet Light",
+ "files.autoSave": "afterDelay",
+ "workbench.colorCustomizations": {
+ "editor.wordHighlightBackground": "#063ef7",
+ // "editor.selectionBackground": "#d1d1c6",
+ // "tab.activeBackground": "#b8b9988c",
+ "editor.selectionHighlightBackground": "#c5293e"
+ },
+```
+
+### Clang format
+
+Now gluten uses clang-format 12 to format source files.
+
+```bash
+apt-get install clang-format-12
+```
+
+Set config in `settings.json`
+
+```json
+"clang-format.executable": "clang-format-12",
+"editor.formatOnSave": true,
+```
+
+If exists multiple clang-format version, formatOnSave may not take effect, specify the default formatter
+Search `default formatter` in `Settings`, select Clang-Format.
+
+If your formatOnSave still make no effect, you can use shortcut `SHIFT+ALT+F` to format one file mannually.
+
+# Debug cpp code with coredump
+
+```bash
+mkdir -p /mnt/DP_disk1/core
+sysctl -w kernel.core_pattern=/mnt/DP_disk1/core/core-%e-%p-%t
+cat /proc/sys/kernel/core_pattern
+# set the core file to unlimited size
+echo "ulimit -c unlimited" >> ~/.bashrc
+# then you will get the core file at `/mnt/DP_disk1/core` when the program crashes
+# gdb -c corefile
+# gdb <gluten_home>/cpp/build/releases/libgluten.so 'core-Executor task l-2000883-1671542526'
+```
+
+'core-Executor task l-2000883-1671542526' is the generated core file name.
+
+```bash
+(gdb) bt
+(gdb) f7
+(gdb) set print pretty on
+(gdb) p *this
+```
+
+- Get the backtrace
+- Switch to 7th stack
+- Print the variable in a more readable way
+- Print the variable fields
+
+Sometimes you only get the cpp exception message, you can generate core dump file by the following code:
+```cpp
+char* p = nullptr;
+*p = 'a';
+```
+or by the following commands:
+- `gcore <pid>`
+- `kill -s SIGSEGV <pid>`
+
+# Debug cpp with gdb
+
+You can use gdb to debug tests and benchmarks.
+And also you can debug jni call.
+Place the following code to your debug path.
+
+```cpp
+pid_t pid = getpid();
+printf("----------------------------------pid: %lun", pid);
+sleep(10);
+```
+
+You can also get the pid by java command or grep java program when executing unit test.
+
+```bash
+jps
+1375551 ScalaTestRunner
+ps ux | grep TestOperator
+```
+
+Execute gdb command to debug:
+```bash
+gdb attach <pid>
+```
+
+```bash
+gdb attach 1375551
+wait to attach....
+(gdb) b <velox_home>/velox/substrait/SubstraitToVeloxPlan.cpp:577
+(gdb) c
+```
+
+# Run TPC-H and TPC-DS
+
+We supply `<gluten_home>/tools/gluten-it` to execute these queries
+Refer to [velox_be.yml](https://github.com/oap-project/gluten/blob/main/.github/workflows/velox_be.yml)
+
+# Run gluten+velox on clean machine
+
+We can run gluten + velox on clean machine by one command (supported OS: Ubuntu20.04/22.04, Centos 7/8, etc.).
+```
+spark-shell --name run_gluten \
+ --master yarn --deploy-mode client \
+ --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=20g \
+ --jars https://github.com/oap-project/gluten/releases/download/v1.0.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.0.0.jar \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
+```
diff --git a/archives/v1.1.1/developers/SubstraitModifications.md b/archives/v1.1.1/developers/SubstraitModifications.md
new file mode 100644
index 0000000..bcedade
--- /dev/null
+++ b/archives/v1.1.1/developers/SubstraitModifications.md
@@ -0,0 +1,37 @@
+---
+layout: page
+title: v1.1.1/Substrait Modifications
+nav_order: 9
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/substrait/
+---
+# Substrait Modifications in Gluten
+
+Substrait is a project aiming to create a well-defined, cross-language specification for data compute operations.
+Since it is still under active development, there are some lacking representations for Gluten needed computing
+operations. At the same time, some existing representations need to be modified a bit to satisfy the needs of computing.
+
+
+In Gluten, the base version of Substrait is `v0.23.0`. This page records all the Gluten changes to Substrait proto
+files for reference. It is preferred to upstream these changes to Substrait, but for those cannot be upstreamed,
+alternatives like `AdvancedExtension` could be considered.
+
+## Modifications to algebra.proto
+
+* Added `JsonReadOptions` and `TextReadOptions` in `FileOrFiles`([#1584](https://github.com/oap-project/gluten/pull/1584)).
+* Changed join type `JOIN_TYPE_SEMI` to `JOIN_TYPE_LEFT_SEMI` and `JOIN_TYPE_RIGHT_SEMI`([#408](https://github.com/oap-project/gluten/pull/408)).
+* Added `WindowRel`, added `column_name` and `window_type` in `WindowFunction`,
+changed `Unbounded` in `WindowFunction` into `Unbounded_Preceding` and `Unbounded_Following`, and added WindowType([#485](https://github.com/oap-project/gluten/pull/485)).
+* Added `output_schema` in RelRoot([#1901](https://github.com/oap-project/gluten/pull/1901)).
+* Added `ExpandRel`([#1361](https://github.com/oap-project/gluten/pull/1361)).
+* Added `GenerateRel`([#574](https://github.com/oap-project/gluten/pull/574)).
+* Added `PartitionColumn` in `LocalFiles`([#2405](https://github.com/oap-project/gluten/pull/2405)).
+* Added `WriteRel` ([#3690](https://github.com/oap-project/gluten/pull/3690)).
+
+## Modifications to type.proto
+
+* Added `Nothing` in `Type`([#791](https://github.com/oap-project/gluten/pull/791)).
+* Added `names` in `Struct`([#1878](https://github.com/oap-project/gluten/pull/1878)).
+* Added `PartitionColumns` in `NamedStruct`([#320](https://github.com/oap-project/gluten/pull/320)).
+* Remove `PartitionColumns` and add `column_types` in `NamedStruct`([#2405](https://github.com/oap-project/gluten/pull/2405)).
diff --git a/archives/v1.1.1/developers/UsingGperftoolsInCH.md b/archives/v1.1.1/developers/UsingGperftoolsInCH.md
new file mode 100644
index 0000000..f0d5c72
--- /dev/null
+++ b/archives/v1.1.1/developers/UsingGperftoolsInCH.md
@@ -0,0 +1,26 @@
+We need using gpertools to find the memory or CPU issue. That's what this document is about.
+
+## Install gperftools
+Install gperftools as described in https://github.com/gperftools/gperftools.
+We get the library and the command line tools.
+
+## Compiler libch.so
+Disable jemalloc `-DENABLE_JEMALLOC=OFF` in cpp-ch/CMakeLists.txt, and recompile libch.so.
+
+## Run Gluten with gperftools
+For Spark on Yarn, we can change the submit script to run Gluten with gperftools.
+Add the following to the submit script:
+```
+export tcmalloc_path=/data2/zzb/gperftools-2.10/.libs/libtcmalloc_and_profiler.so # the path to the tcmalloc library
+export LD_PRELOAD=$tcmalloc_path,libch.so # load the library in the driver
+--files $tcmalloc_path # upload the library to the cluster
+--conf spark.executorEnv.LD_PRELOAD=./libtcmalloc_and_profiler.so,libch.so # load the library in the executor
+--conf spark.executorEnv.HEAPPROFILE=/tmp/gluten_heap_perf # set the heap profile path, you can change to CPUPROFILE for CPU profiling
+```
+
+For thrift server on local machine, note using `export LD_PRELOAD="$tcmalloc_path libch.so" # load the library in the driver` to preload dynamic libraries.
+
+## Analyze the result
+We can get the result in the path we set in the previous step. For example, we can get the result in `/tmp/gluten_heap_perf`. We can use the following website to analyze the result:
+https://gperftools.github.io/gperftools/heapprofile.html
+https://gperftools.github.io/gperftools/cpuprofile.html
diff --git a/archives/v1.1.1/developers/UsingJemallocWithCH.md b/archives/v1.1.1/developers/UsingJemallocWithCH.md
new file mode 100644
index 0000000..626f752
--- /dev/null
+++ b/archives/v1.1.1/developers/UsingJemallocWithCH.md
@@ -0,0 +1,40 @@
+We need using jemalloc to find the memory issue. That's what this document is about.
+
+## Change code of jemalloc
+Use libunwind instead of libgcc to get the backtrace which may cause process hangs.
+```
+@@ -177,7 +177,8 @@ target_compile_definitions(_jemalloc PRIVATE -DJEMALLOC_PROF=1)
+ # At the time ClickHouse uses LLVM libunwind which follows libgcc's way of backtracking.
+ #
+ # ClickHouse has to provide `unw_backtrace` method by the means of [commit 8e2b31e](https://github.com/ClickHouse/libunwind/commit/8e2b31e766dd502f6df74909e04a7dbdf5182eb1).
+-target_compile_definitions (_jemalloc PRIVATE -DJEMALLOC_PROF_LIBGCC=1)
++#target_compile_definitions (_jemalloc PRIVATE -DJEMALLOC_PROF_LIBGCC=1)
++target_compile_definitions (_jemalloc PRIVATE -DJEMALLOC_PROF_LIBUNWIND=1)
+ target_link_libraries (_jemalloc PRIVATE unwind)
+ ```
+For more infomation, check https://github.com/jemalloc/jemalloc/issues/2282.
+
+## Get jeprof
+Change to the directory where you want to install jemalloc, and run the following commands:
+```
+cd $Clickhouse_SOURCE_PATH/contrib/jemalloc && ./autogen.sh && ./configure.sh && make -j8
+```
+Then we get jeprof in the directory `$Clickhouse_SOURCE_PATH/contrib/jemalloc/bin/jeprof`.
+
+## Compiler libch.so
+Ensure to enable jemalloc `-DENABLE_JEMALLOC=ON` in cpp-ch/CMakeLists.txt, and compile libch.so.
+
+## Run Gluten with jemalloc heap tools
+For Yarn or thrift server, you need add the following to the submit script:
+```
+export MALLOC_CONF="prof:true,lg_prof_interval:30" # enable jemalloc heap profiling
+```
+You can find more options in https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling.
+
+## Analyze the result
+After you get the heap file, you can use the following command to analyze the result:
+Check memory diff, so you can find which part of the code consume the memory.
+```
+jeprof --svg /usr/lib/jvm/java-8-openjdk-amd64/bin/java --base=jeprof.3717358.1.i1.heap jeprof.3717358.30.i30.heap > diff.svg
+```
+You can find more usage about jeprof with `jeprof --help`.
\ No newline at end of file
diff --git a/archives/v1.1.1/developers/docker_centos7.md b/archives/v1.1.1/developers/docker_centos7.md
new file mode 100644
index 0000000..b1f726a
--- /dev/null
+++ b/archives/v1.1.1/developers/docker_centos7.md
@@ -0,0 +1,54 @@
+---
+layout: page
+title: v1.1.1/Docker script for CentOS 7
+nav_order: 8
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/docker_centos7/
+---
+Here is a docker script we verified to build Gluten+Velox backend on CentOS 7:
+
+Run on host as root user:
+```
+docker pull centos:7
+docker run -itd --name gluten centos:7 /bin/bash
+docker attach gluten
+```
+
+Run in docker:
+```
+yum -y install epel-release centos-release-scl
+yum -y install \
+ git \
+ dnf \
+ cmake3 \
+ devtoolset-9 \
+ java-1.8.0-openjdk \
+ java-1.8.0-openjdk-devel \
+ ninja-build \
+ wget \
+ sudo
+
+# gluten need maven version >=3.6.3
+wget https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
+tar -xvf apache-maven-3.8.8-bin.tar.gz
+mv apache-maven-3.8.8 /usr/lib/maven
+export MAVEN_HOME=/usr/lib/maven
+export PATH=${PATH}:${MAVEN_HOME}/bin
+
+# cmake 3.x is required
+ln -s /usr/bin/cmake3 /usr/local/bin/cmake
+
+# enable gcc 9
+. /opt/rh/devtoolset-9/enable || exit 1
+
+export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+
+git clone https://github.com/oap-project/gluten.git
+cd gluten
+
+# To access HDFS or S3, you need to add the parameters `--enable_hdfs=ON` and `--enable_s3=ON`
+# If you have the same error with issue-3283, you need to add the parameter `--compile_arrow_java=ON`
+./dev/buildbundle-veloxbe.sh
+```
diff --git a/archives/v1.1.1/developers/docker_centos8.md b/archives/v1.1.1/developers/docker_centos8.md
new file mode 100755
index 0000000..8eaa04d
--- /dev/null
+++ b/archives/v1.1.1/developers/docker_centos8.md
@@ -0,0 +1,51 @@
+---
+layout: page
+title: v1.1.1/Docker script for CentOS 8
+nav_order: 7
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/docker_centos8
+---
+Here is a docker script we verified to build Gluten+Velox backend on Centos8:
+
+Run on host as root user:
+```
+docker pull centos:8
+docker run -itd --name gluten centos:8 /bin/bash
+docker attach gluten
+```
+
+Run in docker:
+```
+#update mirror
+sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-*
+sed -i -e "s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g" /etc/yum.repos.d/CentOS-*
+
+dnf install -y epel-release sudo
+yum install -y dnf-plugins-core
+yum config-manager --set-enabled powertools
+dnf --enablerepo=powertools install -y ninja-build
+dnf --enablerepo=powertools install -y libdwarf-devel
+dnf install -y --setopt=install_weak_deps=False ccache gcc-toolset-9 git wget which libevent-devel \
+ openssl-devel re2-devel libzstd-devel lz4-devel double-conversion-devel \
+ curl-devel cmake libicu-devel
+
+source /opt/rh/gcc-toolset-9/enable || exit 1
+
+yum install -y java-1.8.0-openjdk-devel patch
+export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
+export PATH=$JAVA_HOME/bin:$PATH
+
+#gluten need maven version >=3.6.3
+wget https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
+tar -xvf apache-maven-3.8.8-bin.tar.gz
+mv apache-maven-3.8.8 /usr/lib/maven
+export MAVEN_HOME=/usr/lib/maven
+export PATH=${PATH}:${MAVEN_HOME}/bin
+
+git clone https://github.com/oap-project/gluten.git
+cd gluten
+
+# To access HDFS or S3, you need to add the parameters `--enable_hdfs=ON` and `--enable_s3=ON`
+./dev/buildbundle-veloxbe.sh
+```
diff --git a/archives/v1.1.1/developers/docker_ubuntu22.04.md b/archives/v1.1.1/developers/docker_ubuntu22.04.md
new file mode 100644
index 0000000..8186607
--- /dev/null
+++ b/archives/v1.1.1/developers/docker_ubuntu22.04.md
@@ -0,0 +1,63 @@
+---
+layout: page
+title: v1.1.1/Docker script for Ubuntu 22.04/20.04
+nav_order: 6
+parent: Developers
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/docker_ubuntu/
+---
+
+To the first build, it's suggested to build Gluten in a clean docker image. Otherwise it's easy to run into library version conflict issues.
+Here is a docker script we verified to build Gluten+Velox backend on Ubuntu22.04/20.04:
+
+Run on host as root user:
+```
+docker pull ubuntu:22.04
+docker run -itd --network host --name gluten ubuntu:22.04 /bin/bash
+docker attach gluten
+```
+
+Run in docker:
+```
+apt-get update
+
+#install gcc and libraries to build arrow
+apt install software-properties-common
+apt install maven build-essential cmake libssl-dev libre2-dev libcurl4-openssl-dev clang lldb lld libz-dev git ninja-build uuid-dev autoconf-archive curl zip unzip tar pkg-config bison libtool flex vim
+
+#velox script needs sudo to install dependency libraries
+apt install sudo
+
+# make sure jemalloc is uninstalled, jemalloc will be build in vcpkg, which conflicts with the default jemalloc in system
+apt purge libjemalloc-dev libjemalloc2 librust-jemalloc-sys-dev
+
+#make sure jdk8 is used. New version of jdk is not supported
+apt install -y openjdk-8-jdk
+apt install -y default-jdk
+export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+export PATH=$JAVA_HOME/bin:$PATH
+
+#manually install tzdata to avoid the interactive timezone config
+ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
+DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata
+dpkg --configure -a
+
+#setup proxy on necessary
+#export http_proxy=xxxx
+#export https_proxy=xxxx
+
+#clone gluten
+git clone https://github.com/oap-project/gluten.git
+cd gluten/
+
+#config maven proxy
+#mkdir ~/.m2/
+#vim ~/.m2/settings.xml
+
+# the script download velox & arrow and compile all dependency library automatically
+# To access HDFS or S3, you need to add the parameters `--enable_hdfs=ON` and `--enable_s3=ON`
+# It's suggested to build using static link, enabled by `--enable_vcpkg=ON`
+# For developer, it's suggested to enable Debug info, by --build_type=RelWithDebInfo. Note RelWithDebInfo uses -o2, release uses -o3
+./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON --build_type=RelWithDebInfo
+```
diff --git a/archives/v1.1.1/developers/index.md b/archives/v1.1.1/developers/index.md
new file mode 100644
index 0000000..c1614ec
--- /dev/null
+++ b/archives/v1.1.1/developers/index.md
@@ -0,0 +1,12 @@
+---
+layout: page
+title: v1.1.1/Developers
+nav_order: 3
+has_children: true
+parent: v1.1.1
+grand_parent: Archives
+permalink: /archives/v1.1.1/developers/
+---
+# Gluten Developers
+This document provides a developer overview of the project and covers the
+following topics:
diff --git a/archives/v1.1.1/developers/velox-function-development-guide.md b/archives/v1.1.1/developers/velox-function-development-guide.md
new file mode 100644
index 0000000..dabd8a6
--- /dev/null
+++ b/archives/v1.1.1/developers/velox-function-development-guide.md
@@ -0,0 +1,48 @@
+---
+layout: page
+title: v1.1.1/Velox Function Development
+nav_order: 5
+parent: Developers
+parent: v1.1.1/Developers
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/developers/velox_function_development/
+---
+# Developer Guide for Implementing Spark Built-in SQL Functions in Velox
+
+In velox, two folders `prestosql` & `sparksql` are holding most sql functions, respective for `presto` and `spark`. Gluten will ask velox to firstly register `prestosql` functions, then `sparksql` functions. So if `prestosql`
+and `sparksql` share same signature for a function, the `sparksql` function will overwrite the corresponding `prestosql` function. If the required function is lacking in both folders (exceptions are some common functions defined
+outside, like `cast`), we need to implement the missing function in `sparksql` folder. It is possible that a `prestosql` function has some semantic difference with the corresponding spark function, even though they share the
+same name and function signature. If so, we also need to do an implementation in `sparksql` folder, generally based on the original impl. for `prestosql`.
+
+There are a few spark functions that can behave differently for some special cases, depending on ANSI on or off. Currently, gluten does NOT support ANSI mode. So only ANSI off needs to be considered in implementing spark
+built-in functions in velox.
+
+Take `BitwiseAndFunction` as example:
+
+```
+template <typename T>
+struct BitwiseAndFunction {
+ template <typename TInput>
+ // For void return type, it indicates null result will never be obtained for non-null input.
+ // For bool return type, it indicates null result can be obtained for non-null input (false for null).
+ FOLLY_ALWAYS_INLINE void call(TInput& result, TInput a, TInput b) {
+ result = a & b;
+ }
+};
+```
+It is templated, as well as the `call` function, to allow multiple types. In the above impl., the result will be null for null input.
+Please use `callNullable` if you need different behavior for null input, e.g., get a non-null result for null input. Also see `callNullFree` in velox document.
+It is used for fast evaluation in the case that any input has null.
+
+The below code will register the implemented function for all kinds of integer types. The specified name `bitwise_and` will be actually used in calling this function.
+```
+registerBinaryIntegral<BitwiseAndFunction>({prefix + "bitwise_and"});
+```
+
+Functions for complex types have similar implementations.
+See `ArrayAverageFunction` in [velox/functions/prestosql/ArrayFunctions.h](https://github.com/facebookincubator/velox/blob/main/velox/functions/prestosql/ArrayFunctions.h).
+
+### Reference:
+Velox’s official developer guide:
+ * [velox/docs/develop/scalar-functions.rst](https://github.com/facebookincubator/velox/blob/main/velox/docs/develop/scalar-functions.rst)
+ * [velox/examples/SimpleFunctions.cpp](https://github.com/facebookincubator/velox/blob/main/velox/examples/SimpleFunctions.cpp)
diff --git a/archives/v1.1.1/docs/ClickHouse.md b/archives/v1.1.1/docs/ClickHouse.md
new file mode 100644
index 0000000..eef6555
--- /dev/null
+++ b/archives/v1.1.1/docs/ClickHouse.md
@@ -0,0 +1,10 @@
+---
+layout: page
+title: v1.1.1/ClickHouse Backend
+nav_order: 2
+has_children: true
+parent: v1.1.1/Documentation
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/docs/clickhouse/
+---
+# Gluten Documents by version
diff --git a/archives/v1.1.1/docs/GettingStarted_ClickHouse.md b/archives/v1.1.1/docs/GettingStarted_ClickHouse.md
new file mode 100644
index 0000000..fbac41d
--- /dev/null
+++ b/archives/v1.1.1/docs/GettingStarted_ClickHouse.md
@@ -0,0 +1,702 @@
+---
+layout: page
+title: v1.1.1/Getting Started with ClickHouse Backend
+nav_order: 1
+parent: v1.1.1/ClickHouse Backend
+grand_parent: v1.1.1/Documentation
+permalink: /archives/v1.1.1/docs/clickhouse/getting-started/
+---
+## ClickHouse Backend
+
+ClickHouse is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP), which supports best in the industry query performance, while significantly reducing storage requirements through its innovative use of columnar storage and compression.
+We port ClickHouse ( based on version **23.1** ) as a library, called 'libch.so', and Gluten loads this library through JNI as the native engine. In this way, we don't need to deploy a standalone ClickHouse Cluster, Spark uses Gluten as SparkPlugin to read and write ClickHouse MergeTree data.
+
+### Architecture
+
+The architecture of the ClickHouse backend is shown below:
+
+
+
+1. On Spark driver, Spark uses Gluten SparkPlugin to transform the physical plan to the Substrait plan, and then pass the Substrait plan to ClickHouse backend through JNI call on executors.
+2. Based on Spark DataSource V2 interface, implementing a ClickHouse Catalog to support operating the ClickHouse tables, and then using Delta to save some metadata about ClickHouse like the MergeTree parts information, and also provide ACID transactions.
+3. When querying from a ClickHouse table, it will fetch MergeTree parts information from Delta metadata and assign these parts into Spark partitions according to some strategies.
+4. When writing data into a ClickHouse table, it will use ClickHouse library to write MergeTree parts data and collect these MergeTree parts information after writing successfully, and then save these MergeTree parts information into Delta metadata. ( **The feature of writing MergeTree parts is coming soon.** )
+5. On Spark executors, each executor will load the 'libch.so' through JNI when starting, and then call the operators according to the Substrait plan which is passed from Spark Driver, like reading data from the MergeTree parts, writing the MergeTree parts, filtering data, aggregating data and so on.
+6. Currently, the ClickHouse backend only supports reading the MergeTree parts from local storage, it needs to use a high-performance shared file system to share a root bucket on every node of the cluster from the object storage, like JuiceFS.
+
+
+### Development environment setup
+
+In general, we use IDEA for Gluten development and CLion for ClickHouse backend development on **Ubuntu 20**.
+
+#### Prerequisites
+
+Install the software required for compilation, run `sudo ./ep/build-clickhouse/src/install_ubuntu.sh`.
+Under the hood, it will install the following software:
+- Clang 16.0
+- cmake 3.20 or higher version
+- ninja-build 1.8.2
+
+You can also refer to [How-to-Build-ClickHouse-on-Linux](https://clickhouse.com/docs/en/development/build/).
+You need to install the following software manually:
+- Java 8
+- Maven 3.6.3 or higher version
+- Spark 3.2.2 or Spark 3.3.1
+
+Then, get Gluten code:
+```
+ git clone https://github.com/oap-project/gluten.git
+```
+
+#### Setup ClickHouse backend development environment
+
+If you don't care about development environment, you can skip this part.
+
+Otherwise, do:
+
+1. clone Kyligence/ClickHouse repo
+ ```
+ cd /to/some/place/
+ git clone --recursive --shallow-submodules -b clickhouse_backend https://github.com/Kyligence/ClickHouse.git
+ ```
+
+2. Configure cpp-ch
+ ${GLUTEN_SOURCE}/cpp-ch can be treated as an add-on of Kyligence/Clickhouse
+
+ First, initialize some configuration for this add-on:
+
+ ```shell
+ export GLUTEN_SOURCE=/path/to/gluten
+ export CH_SOURCE_DIR=/path/to/ClickHouse
+ cmake -G Ninja -S ${GLUTEN_SOURCE}/cpp-ch -B ${GLUTEN_SOURCE}/cpp-ch/build_ch -DCH_SOURCE_DIR=${CH_SOURCE_DIR} "-DCMAKE_C_COMPILER=$(command -v clang-16)" "-DCMAKE_CXX_COMPILER=$(command -v clang++-16)" "-DCMAKE_BUILD_TYPE=RelWithDebInfo"
+ ```
+
+ Next, you need to compile Kyligence/Clickhouse. There are two options:
+
+3. (Option 1) Use CLion
+
+ - Open ClickHouse repo
+ - Choose File -> Settings -> Build, Execution, Deployment -> Toolchains, and then choose Bundled CMake, clang-16 as C Compiler, clang++-16 as C++ Compiler:
+
+ 
+
+ - Choose File -> Settings -> Build, Execution, Deployment -> CMake:
+
+ 
+
+ And then add these options into CMake options:
+ ```
+ -DENABLE_PROTOBUF=ON -DENABLE_TESTS=OFF -DENABLE_JEMALLOC=ON -DENABLE_MULTITARGET_CODE=ON -DENABLE_EXTERN_LOCAL_ENGINE=ON
+ ```
+ - Build 'ch' target on ClickHouse Project with Debug mode or Release mode:
+
+ 
+
+ If it builds with Release mode successfully, there is a library file called 'libch.so' in path '${CH_SOURCE_DIR}/cmake-build-release/utils/extern-local-engine/'.
+
+ If it builds with Debug mode successfully, there is a library file called 'libchd.so' in path '${CH_SOURCE_DIR}/cmake-build-debug/utils/extern-local-engine/'.
+
+4. (Option 2) Use command line
+ ```
+ cmake --build ${GLUTEN_SOURCE}/cpp-ch/build_ch --target build_ch
+ ```
+ If it builds successfully, there is a library file called 'libch.so' in path '${GLUTEN_SOURCE}/cpp-ch/build/utils/extern-local-engine/'.
+
+### Directly Compile ClickHouse backend
+
+In case you don't want a develop environment, you can use the following command to compile ClickHouse backend directly:
+
+```
+git clone https://github.com/oap-project/gluten.git
+cd gluten
+bash ./ep/build-clickhouse/src/build_clickhouse.sh
+```
+
+This will download Clickhouse for you and build everything.
+The target file is `/path/to/gluten/cpp-ch/build/utils/extern-local-engine/libch.so`.
+
+
+
+### Compile Gluten
+
+The prerequisites are the same as the one mentioned above. Compile Gluten with ClickHouse backend through maven:
+
+- for Spark 3.2.2<span id="deploy-spark-322"></span>
+
+```
+ git clone https://github.com/oap-project/gluten.git
+ cd gluten/
+ export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
+ mvn clean install -Pbackends-clickhouse -Phadoop-2.7.4 -Pspark-3.2 -Dhadoop.version=2.8.5 -DskipTests -Dcheckstyle.skip
+ ls -al backends-clickhouse/target/gluten-XXXXX-spark-3.2-jar-with-dependencies.jar
+```
+
+- for Spark 3.3.1
+
+```
+ git clone https://github.com/oap-project/gluten.git
+ cd gluten/
+ export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g"
+ mvn clean install -Pbackends-clickhouse -Phadoop-2.7.4 -Pspark-3.3 -Dhadoop.version=2.8.5 -DskipTests -Dcheckstyle.skip
+ ls -al backends-clickhouse/target/gluten-XXXXX-spark-3.3-jar-with-dependencies.jar
+```
+
+### Gluten in local Spark Thrift Server
+
+#### Prepare working directory
+
+- for Spark 3.2.2
+
+```
+tar zxf spark-3.2.2-bin-hadoop2.7.tgz
+cd spark-3.2.2-bin-hadoop2.7
+rm -f jars/protobuf-java-2.5.0.jar
+#download protobuf-java-3.23.4.jar, delta-core_2.12-2.0.1.jar and delta-storage-2.0.1.jar
+wget https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.23.4/protobuf-java-3.23.4.jar -P ./jars
+wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.0.1/delta-core_2.12-2.0.1.jar -P ./jars
+wget https://repo1.maven.org/maven2/io/delta/delta-storage/2.0.1/delta-storage-2.0.1.jar -P ./jars
+cp gluten-XXXXX-spark-3.2-jar-with-dependencies.jar jars/
+```
+
+- for Spark 3.3.1
+
+```
+tar zxf spark-3.3.1-bin-hadoop2.7.tgz
+cd spark-3.3.1-bin-hadoop2.7
+rm -f jars/protobuf-java-2.5.0.jar
+#download protobuf-java-3.23.4.jar, delta-core_2.12-2.2.0.jar and delta-storage-2.2.0.jar
+wget https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.23.4/protobuf-java-3.23.4.jar -P ./jars
+wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.2.0/delta-core_2.12-2.2.0.jar -P ./jars
+wget https://repo1.maven.org/maven2/io/delta/delta-storage/2.2.0/delta-storage-2.2.0.jar -P ./jars
+cp gluten-XXXXX-spark-3.3-jar-with-dependencies.jar jars/
+```
+
+#### Query local data
+
+##### Start Spark Thriftserver on local
+```
+cd spark-3.2.2-bin-hadoop2.7
+./sbin/start-thriftserver.sh \
+ --master local[3] \
+ --driver-memory 10g \
+ --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.default.parallelism=1 \
+ --conf spark.sql.shuffle.partitions=1 \
+ --conf spark.sql.files.minPartitionNum=1 \
+ --conf spark.sql.files.maxPartitionBytes=1073741824 \
+ --conf spark.sql.adaptive.enabled=false \
+ --conf spark.locality.wait=0 \
+ --conf spark.locality.wait.node=0 \
+ --conf spark.locality.wait.process=0 \
+ --conf spark.sql.columnVector.offheap.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=6442450944 \
+ --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.gluten.sql.columnar.columnarToRow=true \
+ --conf spark.executorEnv.LD_PRELOAD=/path_to_clickhouse_library/libch.so\
+ --conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.io.compression.codec=snappy \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.execution.datasources.v2.clickhouse.ClickHouseSparkCatalog \
+ --conf spark.databricks.delta.maxSnapshotLineageLength=20 \
+ --conf spark.databricks.delta.snapshotPartitions=1 \
+ --conf spark.databricks.delta.properties.defaults.checkpointInterval=5 \
+ --conf spark.databricks.delta.stalenessLimit=3600000
+
+#connect to Spark Thriftserver by beeline
+bin/beeline -u jdbc:hive2://localhost:10000/ -n root
+```
+
+##### Query local MergeTree files
+
+- Prepare data<span id="data-preparation"></span>
+
+Currently, the feature of writing ClickHouse MergeTree parts by Spark is developing, so you need to use command 'clickhouse-local' to generate MergeTree parts data manually. We provide a python script to call the command 'clickhouse-local' to convert parquet data to MergeTree parts:
+
+```
+
+#install ClickHouse community version
+sudo apt-get install -y apt-transport-https ca-certificates dirmngr
+sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
+echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
+sudo apt-get update
+sudo apt install -y --allow-downgrades clickhouse-server=22.5.1.2079 clickhouse-client=22.5.1.2079 clickhouse-common-static=22.5.1.2079
+
+#generate MergeTree parts
+mkdir -p /path_clickhouse_database/table_path/
+python3 /path_to_clickhouse_backend_src/utils/local-engine/tool/parquet_to_mergetree.py --path=/tmp --source=/path_to_parquet_data/tpch-data-sf100/lineitem --dst=/path_clickhouse_database/table_path/lineitem
+```
+
+**This python script will convert one parquet data file to one MergeTree parts.**
+
+- Create a TPC-H lineitem table using ClickHouse DataSource
+
+```
+ DROP TABLE IF EXISTS lineitem;
+ CREATE TABLE IF NOT EXISTS lineitem (
+ l_orderkey bigint,
+ l_partkey bigint,
+ l_suppkey bigint,
+ l_linenumber bigint,
+ l_quantity double,
+ l_extendedprice double,
+ l_discount double,
+ l_tax double,
+ l_returnflag string,
+ l_linestatus string,
+ l_shipdate date,
+ l_commitdate date,
+ l_receiptdate date,
+ l_shipinstruct string,
+ l_shipmode string,
+ l_comment string)
+ USING clickhouse
+ TBLPROPERTIES (engine='MergeTree'
+ )
+ LOCATION '/path_clickhouse_database/table_path/lineitem';
+```
+
+- TPC-H Q6 test
+
+```
+ SELECT
+ sum(l_extendedprice * l_discount) AS revenue
+ FROM
+ lineitem
+ WHERE
+ l_shipdate >= date'1994-01-01'
+ AND l_shipdate < date'1994-01-01' + interval 1 year
+ AND l_discount BETWEEN 0.06 - 0.01 AND 0.06 + 0.01
+ AND l_quantity < 24;
+```
+
+- Result
+
+ The DAG is shown on Spark UI as below:
+
+ 
+
+##### Query local Parquet files
+
+You can query local parquet files directly.
+```sql
+-- query on a single file
+select * from parquet.`/your_data_root_dir/1.parquet`;
+
+-- query on a directly which has multiple files
+select * from parquet.`/your_data_roo_dir/`;
+```
+
+You can also create a TEMPORARY VIEW for parquet files.
+```sql
+create or replace temporary view your_table_name
+using org.apache.spark.sql.parquet
+options(
+ path "/your_data_root_dir/"
+)
+```
+
+
+##### Query Parquet files in S3
+
+If you have parquet files in S3(either AWS S3 or S3 compatible storages like MINIO), you can query them directly.
+You need to add these additional configs to spark:
+
+```bash
+ --config spark.hadoop.fs.s3a.endpoint=S3_ENDPOINT
+ --config spark.hadoop.fs.s3a.path.style.access=true
+ --config spark.hadoop.fs.s3a.access.key=YOUR_ACCESS_KEY
+ --config spark.hadoop.fs.s3a.secret.key=YOUR_SECRET_KEY
+```
+where S3_ENDPOINT must follow the format of `https://s3.region-code.amazonaws.com`, e.g. `https://s3.us-east-1.amazonaws.com` (or `http://hostname:39090 for MINIO)
+
+When you query the parquet files in S3, you need to add the prefix `s3a://` to the path, e.g. `s3a://your_bucket_name/path_to_your_parquet`.
+
+Additionally, you can add these configs to enable local caching of S3 data. Each spark executor will have its own cache. Cache stealing between executors is not supported yet.
+
+```bash
+ --config spark.gluten.sql.columnar.backend.ch.runtime_config.s3.local_cache.enabled=true
+ --config spark.gluten.sql.columnar.backend.ch.runtime_config.s3.local_cache.cache_path=/executor_local_folder_for_cache
+```
+
+##### Use beeline to execute queries
+
+After start a spark thriftserver, we can use the beeline to connect to this server.
+```bash
+# run a file
+/path_to_spark/bin/beeline -u jdbc:hive2://localhost:10000 -f <your_sql_file>
+
+# run a query
+/path_to_spark/bin/beeline -u jdbc:hive2://localhost:10000 -e '<your_sql>'
+
+# enter a interactive mode
+/path_to_spark/bin/beeline -u jdbc:hive2://localhost:10000
+```
+
+#### Query Hive tables in HDFS
+
+Suppose that you have set up hive and hdfs, you can query the data on hive directly.
+
+- Copy `hive-site.xml` into `/path_to_spark/conf/`
+- Copy `hdfs-site.xml` into `/path_to_spark/conf/`, and edit `spark-env.sh`
+
+```bash
+# add this line into spark-env.sh
+export HADOOP_CONF_DIR=/path_to_spark/conf
+```
+
+- Start spark thriftserver with hdfs configurations
+
+```bash
+hdfs_conf_file=/your_local_path/hdfs-site.xml
+
+cd spark-3.2.2-bin-hadoop2.7
+# add a new option: spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf
+./sbin/start-thriftserver.sh \
+ --master local[3] \
+ --files $hdfs_conf_file \
+ --driver-memory 10g \
+ --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.default.parallelism=1 \
+ --conf spark.sql.shuffle.partitions=1 \
+ --conf spark.sql.files.minPartitionNum=1 \
+ --conf spark.sql.files.maxPartitionBytes=1073741824 \
+ --conf spark.locality.wait=0 \
+ --conf spark.locality.wait.node=0 \
+ --conf spark.locality.wait.process=0 \
+ --conf spark.sql.columnVector.offheap.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=6442450944 \
+ --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.gluten.sql.columnar.columnarToRow=true \
+ --conf spark.executorEnv.LD_PRELOAD=/path_to_clickhouse_library/libch.so\
+ --conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.io.compression.codec=snappy \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.execution.datasources.v2.clickhouse.ClickHouseSparkCatalog \
+ --conf spark.databricks.delta.maxSnapshotLineageLength=20 \
+ --conf spark.databricks.delta.snapshotPartitions=1 \
+ --conf spark.databricks.delta.properties.defaults.checkpointInterval=5 \
+ --conf spark.databricks.delta.stalenessLimit=3600000 \
+ --conf spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf=./hdfs-site.xml
+```
+
+For example, you have a table `demo_database`.`demo_table` on the hive, you can run queries as below.
+
+```sql
+select * from demo_database.demo_talbe;
+```
+
+
+### Gluten in YARN cluster
+
+We can to run a Spark SQL task by gluten on a yarn cluster as following
+```bash
+#!/bin/bash
+
+# The file contains the sql you want to run
+sql_file=/path/to/spark/sql/file
+
+export SPARK_HOME=/path/to/spark/home
+spark_cmd=$SPARK_HOME/bin/spark-sql
+
+# Define the path to libch.so
+ch_lib=/path/to/libch.so
+export LD_PRELOAD=$ch_lib
+
+# copy gluten jar file to $SPARK_HOME/jar
+gluten_jar=/path/to/gluten/jar/file
+cp $gluten_jar $SPARK_HOME/jar
+
+batchsize=20480
+hdfs_conf=/path/to/hdfs-site.xml
+
+$spark_cmd \
+ --name gluten_on_yarn
+ --master yarn \
+ --deploy-mode client \
+ --files $ch_lib \
+ --executor-cores 1 \
+ --num-executors 2 \
+ --executor-memory 10g \
+ --conf spark.default.parallelism=4 \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=7g \
+ --conf spark.driver.maxResultSize=2g \
+ --conf spark.sql.autoBroadcastJoinThreshold=-1 \
+ --conf spark.sql.parquet.columnarReaderBatchSize=${batchsize} \
+ --conf spark.sql.inMemoryColumnarStorage.batchSize=${batchsize} \
+ --conf spark.sql.execution.arrow.maxRecordsPerBatch=${batchsize} \
+ --conf spark.sql.broadcastTimeout=4800 \
+ --conf spark.task.maxFailures=1 \
+ --conf spark.excludeOnFailure.enabled=false \
+ --conf spark.driver.maxResultSize=4g \
+ --conf spark.sql.adaptive.enabled=false \
+ --conf spark.dynamicAllocation.executorIdleTimeout=0s \
+ --conf spark.sql.shuffle.partitions=112 \
+ --conf spark.sql.sources.useV1SourceList=avro \
+ --conf spark.sql.files.maxPartitionBytes=1073741824 \
+ --conf spark.gluten.sql.columnar.columnartorow=true \
+ --conf spark.gluten.sql.columnar.loadnative=true \
+ --conf spark.gluten.sql.columnar.libpath=$ch_lib \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.gluten.sql.columnar.backend.ch.runtime_config.hdfs.libhdfs3_conf=$hdfs_conf \
+ --conf spark.gluten.sql.columnar.backend.ch.runtime_config.logger.level=debug \
+ --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.executorEnv.LD_PRELOAD=$LD_PRELOAD \
+ --conf spark.hadoop.input.connect.timeout=600000 \
+ --conf spark.hadoop.input.read.timeout=600000 \
+ --conf spark.hadoop.input.write.timeout=600000 \
+ --conf spark.hadoop.dfs.client.log.severity="DEBUG2" \
+ --files $ch_lib \
+ -f $sql_file
+```
+
+We also can use `spark-submit` to run a task.
+
+
+### Benchmark with TPC-H 100 Q6 on Gluten with ClickHouse backend
+
+This benchmark is tested on AWS EC2 cluster, there are 7 EC2 instances:
+
+| Node Role | EC2 Type | Instances Count | Resources | AMI |
+| ---------- | ----------- | ------------- | ------------- | ------------- |
+| Master | m5.4xlarge | 1 | 16 cores 64G memory per node | ubuntu-focal-20.04 |
+| Worker | m5.4xlarge | 1 | 16 cores 64G memory per node | ubuntu-focal-20.04 |
+
+
+#### Deploy on Cloud
+
+- Tested on Spark Standalone cluster, its resources are shown below:
+
+ | | CPU cores | Memory | Instances Count |
+ | ---------- | ----------- | ------------- | ------------- |
+ | Spark Worker | 15 | 60G | 6 |
+
+- Prepare jars
+
+ Refer to [Deploy Spark 3.2.2](#deploy-spark-322)
+
+- Deploy gluten-core-XXXXX-jar-with-dependencies.jar
+
+```
+ #deploy 'gluten-core-XXXXX-jar-with-dependencies.jar' to every node, and then
+ cp gluten-core-XXXXX-jar-with-dependencies.jar /path_to_spark/jars/
+```
+
+- Deploy ClickHouse library
+
+ Deploy ClickHouse library 'libch.so' to every worker node.
+
+
+##### Deploy JuiceFS
+
+- JuiceFS uses Redis to save metadata, install redis firstly:
+
+```
+ wget https://download.redis.io/releases/redis-6.0.14.tar.gz
+ sudo apt install build-essential
+ tar -zxvf redis-6.0.14.tar.gz
+ cd redis-6.0.14
+ make
+ make install PREFIX=/home/ubuntu/redis6
+ cd ..
+ rm -rf redis-6.0.14
+
+ #start redis server
+ /home/ubuntu/redis6/bin/redis-server /home/ubuntu/redis6/redis.conf
+
+```
+- Use JuiceFS to format a S3 bucket and mount a volumn on every node
+
+ Please refer to [The-JuiceFS-Command-Reference](https://juicefs.com/docs/community/command_reference)
+
+```
+ wget https://github.com/juicedata/juicefs/releases/download/v0.17.5/juicefs-0.17.5-linux-amd64.tar.gz
+ tar -zxvf juicefs-0.17.5-linux-amd64.tar.gz
+
+ ./juicefs format --block-size 4096 --storage s3 --bucket https://s3.cn-northwest-1.amazonaws.com.cn/s3-gluten-tpch100/ --access-key "XXXXXXXX" --secret-key "XXXXXXXX" redis://:123456@master-ip:6379/1 gluten-tables
+
+ #mount a volumn on every node
+ ./juicefs mount -d --no-usage-report --no-syslog --attr-cache 7200 --entry-cache 7200 --dir-entry-cache 7200 --buffer-size 500 --prefetch 1 --open-cache 86400 --log /home/ubuntu/juicefs-logs/mount1.log --cache-dir /home/ubuntu/juicefs-cache/ --cache-size 102400 redis://:123456@master-ip:6379/1 /home/ubuntu/gluten/gluten_table
+ #create a directory for lineitem table path
+ mkdir -p /home/ubuntu/gluten/gluten_table/lineitem
+
+```
+
+#### Preparation
+
+Please refer to [Data-preparation](#data-preparation) to generate MergeTree parts data to the lineitem table path: /home/ubuntu/gluten/gluten_table/lineitem.
+
+#### Run Spark Thriftserver
+
+```
+cd spark-3.2.2-bin-hadoop2.7
+./sbin/start-thriftserver.sh \
+ --master spark://master-ip:7070 --deploy-mode client \
+ --driver-memory 16g --driver-cores 4 \
+ --total-executor-cores 90 --executor-memory 60g --executor-cores 15 \
+ --conf spark.driver.memoryOverhead=8G \
+ --conf spark.default.parallelism=90 \
+ --conf spark.sql.shuffle.partitions=1 \
+ --conf spark.sql.files.minPartitionNum=1 \
+ --conf spark.sql.files.maxPartitionBytes=536870912 \
+ --conf spark.sql.parquet.filterPushdown=true \
+ --conf spark.sql.parquet.enableVectorizedReader=true \
+ --conf spark.locality.wait=0 \
+ --conf spark.locality.wait.node=0 \
+ --conf spark.locality.wait.process=0 \
+ --conf spark.sql.columnVector.offheap.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=42949672960 \
+ --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.gluten.sql.columnar.columnarToRow=true \
+ --conf spark.gluten.sql.columnar.libpath=/path_to_clickhouse_library/libch.so \
+ --conf spark.gluten.sql.columnar.iterator=true \
+ --conf spark.gluten.sql.columnar.loadarrow=false \
+ --conf spark.gluten.sql.columnar.hashagg.enablefinal=true \
+ --conf spark.gluten.sql.enable.native.validation=false \
+ --conf spark.io.compression.codec=snappy \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.execution.datasources.v2.clickhouse.ClickHouseSparkCatalog \
+ --conf spark.databricks.delta.maxSnapshotLineageLength=20 \
+ --conf spark.databricks.delta.snapshotPartitions=1 \
+ --conf spark.databricks.delta.properties.defaults.checkpointInterval=5 \
+ --conf spark.databricks.delta.stalenessLimit=3600000
+```
+
+#### Test TPC-H Q6 with JMeter
+
+- Create a lineitem table using clickhouse datasource
+
+```
+ DROP TABLE IF EXISTS lineitem;
+ CREATE TABLE IF NOT EXISTS lineitem (
+ l_orderkey bigint,
+ l_partkey bigint,
+ l_suppkey bigint,
+ l_linenumber bigint,
+ l_quantity double,
+ l_extendedprice double,
+ l_discount double,
+ l_tax double,
+ l_returnflag string,
+ l_linestatus string,
+ l_shipdate date,
+ l_commitdate date,
+ l_receiptdate date,
+ l_shipinstruct string,
+ l_shipmode string,
+ l_comment string)
+ USING clickhouse
+ TBLPROPERTIES (engine='MergeTree'
+ )
+ LOCATION 'file:///home/ubuntu/gluten/gluten_table/lineitem';
+```
+
+- Run TPC-H Q6 test with JMeter
+ 1. Run TPC-H Q6 test 100 times in the first round;
+ 2. Run TPC-H Q6 test 1000 times in the second round;
+
+#### Performance
+
+The performance of Gluten + ClickHouse backend increases by **about 1/3**.
+
+| | 70% | 80% | 90% | 99% | Avg |
+| ---------- | ----------- | ------------- | ------------- | ------------- | ------------- |
+| Spark + Parquet | 590ms | 592ms | 597ms | 609ms | 588ms |
+| Spark + Gluten + ClickHouse backend | 402ms | 405ms | 409ms | 425ms | 399ms |
+
+
+### New CI System
+
+https://opencicd.kyligence.com/job/Gluten/job/gluten-ci/
+public read-only account：gluten/hN2xX3uQ4m
+
+### Celeborn support
+
+Gluten with clickhouse backend has not yet supportted [Celeborn](https://github.com/apache/incubator-celeborn) natively as remote shuffle service using columar shuffle. However, you can still use Celeborn with row shuffle, which means a ColumarBatch will be converted to a row during shuffle.
+Below introduction is used to enable this feature:
+
+First refer to this URL(https://github.com/apache/incubator-celeborn) to setup a celeborn cluster.
+
+Then add the Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
+
+Currently to use Celeborn following configurations are required in `spark-defaults.conf`
+
+```
+spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+spark.shuffle.service.enabled false
+
+# options: hash, sort
+# Hash shuffle writer use (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
+# Sort shuffle writer uses less memory than hash shuffle writer, if your shuffle partition count is large, try to use sort hash writer.
+spark.celeborn.client.spark.shuffle.writer hash
+
+# We recommend setting spark.celeborn.client.push.replicate.enabled to true to enable server-side data replication
+# If you have only one worker, this setting must be false
+# If your Celeborn is using HDFS, it's recommended to set this setting to false
+spark.celeborn.client.push.replicate.enabled true
+
+# Support for Spark AQE only tested under Spark 3
+# we recommend setting localShuffleReader to false to get better performance of Celeborn
+spark.sql.adaptive.localShuffleReader.enabled false
+
+# If Celeborn is using HDFS
+spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
+
+# If you want to use dynamic resource allocation,
+# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+spark.dynamicAllocation.enabled false
+```
+
+#### Celeborn Columnar Shuffle Support
+The native Celeborn support can be enabled by the following configuration
+```
+spark.shuffle.manager=org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
+```
+
+quickly start a celeborn cluster
+```shell
+wget https://dlcdn.apache.org/incubator/celeborn/celeborn-0.3.0-incubating/apache-celeborn-0.3.0-incubating-bin.tgz && \
+tar -zxvf apache-celeborn-0.3.0-incubating-bin.tgz && \
+mv apache-celeborn-0.3.0-incubating-bin/conf/celeborn-defaults.conf.template apache-celeborn-0.3.0-incubating-bin/conf/celeborn-defaults.conf && \
+mv apache-celeborn-0.3.0-incubating-bin/conf/log4j2.xml.template apache-celeborn-0.3.0-incubating-bin/conf/log4j2.xml && \
+mkdir /opt/hadoop && chmod 777 /opt/hadoop && \
+echo -e "celeborn.worker.flusher.threads 4\nceleborn.worker.storage.dirs /tmp\nceleborn.worker.monitor.disk.enabled false" > apache-celeborn-0.3.0-incubating-bin/conf/celeborn-defaults.conf && \
+bash apache-celeborn-0.3.0-incubating-bin/sbin/start-master.sh && bash apache-celeborn-0.3.0-incubating-bin/sbin/start-worker.sh
+```
+
+### Columnar shuffle mode
+We have two modes of columnar shuffle
+1. prefer cache
+2. prefer spill
+
+Switch through the configuration `spark.gluten.sql.columnar.backend.ch.shuffle.preferSpill`, the default is `false`, enable prefer cache shuffle.
+
+In the prefer cache mode, as much memory as possible will be used to cache the shuffle data. When the memory is insufficient,
+spark will actively trigger the memory spill. You can also specify the threshold size through `spark.gluten.sql.columnar.backend.ch.spillThreshold` to Limit memory usage. The default value is `0MB`, which means no limit on memory usage.
+
+
diff --git a/archives/v1.1.1/docs/GettingStarted_Velox.md b/archives/v1.1.1/docs/GettingStarted_Velox.md
new file mode 100644
index 0000000..017ecef
--- /dev/null
+++ b/archives/v1.1.1/docs/GettingStarted_Velox.md
@@ -0,0 +1,917 @@
+---
+layout: page
+title: v1.1.1/Getting-Started with Velox Backend
+nav_order: 1
+parent: v1.1.1/Velox Backend
+grand_parent: v1.1.1/Documentation
+permalink: /archives/v1.1.1/docs/velox/getting-started/
+---
+# Supported Version
+| Type | Version |
+|-------|------------------------------|
+| Spark | 3.2.2, 3.3.1 |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8 |
+| scala | 2.12
+
+Spark3.4.0 support is still WIP. TPCH/DS can pass, UT is not yet passed.
+
+There are pending PRs for jdk11 support.
+
+
+Currently, the mvn script can automatically fetch and build all dependency libraries incluing Velox. Our nightly build still use Velox under oap-project.
+
+# Prerequisite
+
+Currently, Gluten+Velox backend is only tested on **Ubuntu20.04/Ubuntu22.04/Centos8**. Other kinds of OS support are still in progress. The long term goal is to support several
+common OS and conda env deployment.
+
+Gluten builds with Spark3.2.x and Spark3.3.x now but only fully tested in CI with 3.2.2 and 3.3.1. We will add/update supported/tested versions according to the upstream changes.
+
+we need to set up the `JAVA_HOME` env. Currently, **java 8** is required and the support for java 11/17 is not ready.
+
+**For x86_64**
+
+```bash
+## make sure jdk8 is used
+export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+export PATH=$JAVA_HOME/bin:$PATH
+```
+
+**For aarch64**
+
+```bash
+## make sure jdk8 is used
+export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64
+export PATH=$JAVA_HOME/bin:$PATH
+```
+
+**Get gluten**
+
+```bash
+## config maven, like proxy in ~/.m2/settings.xml
+
+## fetch gluten code
+git clone https://github.com/oap-project/gluten.git
+```
+
+# Build Gluten with Velox Backend
+
+It's recommended to use buildbundle-veloxbe.sh to build gluten in one script.
+[Gluten build guide](./build-guide.md) listed the parameters and their default value of build command for your reference.
+
+**For x86_64 build**
+
+```bash
+cd /path/to/gluten
+
+## The script builds two jars for spark 3.2.2 and 3.3.1.
+./dev/buildbundle-veloxbe.sh
+
+## After a complete build, if you need to re-build the project and only some gluten code is changed,
+## you can use the following command to skip building velox and protobuf.
+# ./dev/buildbundle-veloxbe.sh --enable_ep_cache=ON --build_protobuf=OFF
+## If you have the same error with issue-3283, you need to add the parameter `--compile_arrow_java=ON`
+```
+
+**For aarch64 build:**
+
+```bash
+export CPU_TARGET="aarch64"
+
+cd /path/to/gluten
+
+./dev/builddeps-veloxbe.sh
+```
+
+**Build Velox separately**
+
+Scripts under `/path/to/gluten/ep/build-velox/src` provide `get_velox.sh` and `build_velox.sh` to build Velox separately, you could use these scripts with custom repo/branch/location.
+
+Velox provides arrow/parquet lib. Gluten cpp module need a required VELOX_HOME parsed by --velox_home, if you specify custom ep location, make sure these variables be passed correctly.
+
+```bash
+## fetch Velox and compile
+cd /path/to/gluten/ep/build-velox/src/
+## you could use custom ep location by --velox_home=custom_path, make sure specify --velox_home in build_velox.sh too.
+./get_velox.sh
+## make sure specify --velox_home if you have specified it in get_velox.sh.
+./build_velox.sh
+
+## compile Gluten cpp module
+cd /path/to/gluten/cpp
+## if you use custom velox_home, make sure specified here by --velox_home
+./compile.sh --build_velox_backend=ON
+
+## compile Gluten java module and create package jar
+cd /path/to/gluten
+# For spark3.2.x
+mvn clean package -Pbackends-velox -Prss -Pspark-3.2 -DskipTests
+# For spark3.3.x
+mvn clean package -Pbackends-velox -Prss -Pspark-3.3 -DskipTests
+```
+
+notes：The compilation of `Velox` using the script of `build_velox.sh` may fail caused by `oom`, you can prevent this failure by using the user command of `export NUM_THREADS=4` before executing the above scripts.
+
+Once building successfully, the Jar file will be generated in the directory: package/target/\<gluten-jar\> for Spark 3.2.x/Spark 3.3.x.
+
+## Dependency library deployment
+
+With config `enable_vcpkg=ON`, the dependency libraries will be built and statically linked into libvelox.so and libgluten.so, which is packed into the gluten-jar. In this way, only the gluten-jar is needed to add to `spark.<driver|executor>.extraClassPath` and spark will deploy the jar to each worker node. It's better to build the static version using a clean docker image without any extra libraries installed. On host with some libraries like jemalloc installed, the script may crash wit [...]
+
+With config `enable_vcpkg=OFF`, the dependency libraries won't be statically linked, instead the script will install the libraries to system then pack the dependency libraries into another jar named gluten-package-${Maven-artifact-version}.jar. Then you need to add the jar to extraClassPath then set `spark.gluten.loadLibFromJar=true`. Or you already manually deployed the dependency libraries on each worker node. You may find the libraries list from the gluten-package jar.
+
+## HDFS support
+
+Hadoop hdfs support is ready via the [libhdfs3](https://github.com/apache/hawq/tree/master/depends/libhdfs3) library. The libhdfs3 provides native API for Hadoop I/O without the drawbacks of JNI. It also provides advanced authentication like Kerberos based. Please note this library has several dependencies which may require extra installations on Driver and Worker node.
+
+### Build with HDFS support
+To build Gluten with HDFS support, below command is suggested:
+
+```bash
+cd /path/to/gluten
+./dev/buildbundle-veloxbe.sh --enable_hdfs=ON
+```
+
+### Configuration about HDFS support
+
+HDFS uris (hdfs://host:port) will be extracted from a valid hdfs file path to initialize hdfs client, you do not need to specify it explicitly.
+
+libhdfs3 need a configuration file and [example here](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/src/backend/utils/misc/etc/hdfs-client.xml), this file is a bit different from hdfs-site.xml and core-site.xml.
+Download that example config file to local and do some needed modifications to support HA or else, then set env variable like below to use it, or upload it to HDFS to use, more details [here](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/depends/libhdfs3/src/client/Hdfs.cpp#L171-L189).
+
+```
+// Spark local mode
+export LIBHDFS3_CONF="/path/to/hdfs-client.xml"
+
+// Spark Yarn cluster mode
+--conf spark.executorEnv.LIBHDFS3_CONF="/path/to/hdfs-client.xml"
+
+// Spark Yarn cluster mode and upload hdfs config file
+cp /path/to/hdfs-client.xml hdfs-client.xml
+--files hdfs-client.xml
+```
+
+One typical deployment on Spark/HDFS cluster is to enable [short-circuit reading](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html). Short-circuit reads provide a substantial performance boost to many applications.
+
+By default libhdfs3 does not set the default hdfs domain socket path to support HDFS short-circuit read. If this feature is required in HDFS setup, users may need to setup the domain socket path correctly by patching the libhdfs3 source code or by setting the correct config environment. In Gluten the short-circuit domain socket path is set to "/var/lib/hadoop-hdfs/dn_socket" in [build_velox.sh](https://github.com/oap-project/gluten/blob/main/ep/build-velox/src/build_velox.sh) So we need [...]
+
+```
+sudo mkdir -p /var/lib/hadoop-hdfs/
+sudo chown <sparkuser>:<sparkuser> /var/lib/hadoop-hdfs/
+```
+
+You also need to add configuration to the "hdfs-site.xml" as below:
+
+```
+<property>
+ <name>dfs.client.read.shortcircuit</name>
+ <value>true</value>
+</property>
+<property>
+ <name>dfs.domain.socket.path</name>
+ <value>/var/lib/hadoop-hdfs/dn_socket</value>
+</property>
+```
+
+### Kerberos support
+
+Here are two steps to enable kerberos.
+
+- Make sure the hdfs-client.xml contains
+
+```xml
+<property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+</property>
+```
+
+- Specify the environment variable [KRB5CCNAME](https://github.com/apache/hawq/blob/e9d43144f7e947e071bba48871af9da354d177d0/depends/libhdfs3/src/client/FileSystem.cpp#L56) and upload the kerberos ticket cache file
+
+```
+--conf spark.executorEnv.KRB5CCNAME=krb5cc_0000 --files /tmp/krb5cc_0000
+```
+
+The ticket cache file can be found by `klist`.
+
+## AWS S3 support
+
+Velox supports S3 with the open source [AWS C++ SDK](https://github.com/aws/aws-sdk-cpp) and Gluten uses Velox S3 connector to connect with S3.
+A new build option for S3(enable_s3) is added. Below command is used to enable this feature
+
+```
+cd /path/to/gluten
+./dev/buildbundle-veloxbe.sh --enable_s3=ON
+```
+
+Currently there are several ways to asscess S3 in Spark. Please refer [Velox S3](VeloxS3.md) part for more detailed configurations
+
+## Celeborn support
+
+Gluten with velox backend supports [Celeborn](https://github.com/apache/incubator-celeborn) as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/apache/incubator-celeborn) to setup a celeborn cluster.
+
+When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
+- Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-thirdparty-lib-xx-xx.jar
+
+Currently to use Gluten following configurations are required in `spark-defaults.conf`
+
+```
+spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+spark.shuffle.service.enabled false
+
+# options: hash, sort
+# Hash shuffle writer use (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
+# Sort shuffle writer uses less memory than hash shuffle writer, if your shuffle partition count is large, try to use sort hash writer.
+spark.celeborn.client.spark.shuffle.writer hash
+
+# We recommend setting spark.celeborn.client.push.replicate.enabled to true to enable server-side data replication
+# If you have only one worker, this setting must be false
+# If your Celeborn is using HDFS, it's recommended to set this setting to false
+spark.celeborn.client.push.replicate.enabled true
+
+# Support for Spark AQE only tested under Spark 3
+# we recommend setting localShuffleReader to false to get better performance of Celeborn
+spark.sql.adaptive.localShuffleReader.enabled false
+
+# If Celeborn is using HDFS
+spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
+
+# If you want to use dynamic resource allocation,
+# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark.
+spark.dynamicAllocation.enabled false
+```
+
+## DeltaLake Support
+
+Gluten with velox backend supports [DeltaLake](https://delta.io/) table.
+
+### How to use
+
+First of all, compile gluten-delta module by a `delta` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Pdelta -DskipTests
+```
+
+Then, put the additional gluten-delta jar to the class path (usually it's `$SPARK_HOME/jars`).
+The gluten-delta jar is in `gluten-delta/target` directory.
+
+After the two steps, you can query delta table by gluten/velox without scan's fallback.
+
+Gluten with velox backends also support the column mapping of delta tables.
+About column mapping, see more [here](https://docs.delta.io/latest/delta-column-mapping.html).
+
+## Iceberg Support
+
+Gluten with velox backend supports [Iceberg](https://iceberg.apache.org/) table. Currently, only reading COW (Copy-On-Write) tables is supported.
+
+### How to use
+
+First of all, compile gluten-iceberg module by a `iceberg` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests
+```
+
+Then, put the additional gluten-iceberg jar to the class path (usually it's `$SPARK_HOME/jars`).
+The gluten-iceberg jar is in `gluten-iceberg/target` directory.
+
+After the two steps, you can query iceberg table by gluten/velox without scan's fallback.
+
+# Coverage
+Spark3.3 has 387 functions in total. ~240 are commonly used. Velox's functions have two category, Presto and Spark. Presto has 124 functions implemented. Spark has 62 functions. Spark functions are verified to have the same result as Vanilla Spark. Some Presto functions have the same result as Vanilla Spark but some others have different. Gluten prefer to use Spark functions firstly. If it's not in Spark's list but implemented in Presto, we currently offload to Presto one until we noted [...]
+
+> Velox doesn't support [ANSI mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)), so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will fallback to Vanilla Spark.
+
+To identify what can be offloaded in a query and detailed fallback reasons, user can follow below steps to retrieve corresponding logs.
+
+```
+1) Enable Gluten by proper [configuration](https://github.com/oap-project/gluten/blob/main/docs/Configuration.md).
+
+2) Disable Spark AQE to trigger plan validation in Gluten
+spark.sql.adaptive.enabled = false
+
+3) Check physical plan
+sparkSession.sql("your_sql").explain()
+```
+
+With above steps, you will get a physical plan output like:
+
+```
+== Physical Plan ==
+-Execute InsertIntoHiveTable (7)
+ +- Coalesce (6)
+ +- VeloxColumnarToRowExec (5)
+ +- ^ ProjectExecTransformer (3)
+ +- GlutenRowToArrowColumnar (2)
+ +- Scan hive default.extracted_db_pins (1)
+
+```
+
+`GlutenRowToArrowColumnar`/`VeloxColumnarToRowExec` indicates there is a fallback operator before or after it. And you may find fallback reason like below in logs.
+
+```
+native validation failed due to: in ProjectRel, Scalar function name not registered: get_struct_field, called with arguments: (ROW<col_0:INTEGER,col_1:BIGINT,col_2:BIGINT>, INTEGER).
+```
+
+In the above, the symbol `^` indicates a plan is offloaded to Velox in a stage. In Spark DAG, all such pipelined plans (consecutive plans marked with `^`) are plotted
+inside an umbrella node named `WholeStageCodegenTransformer` (It's not codegen node. The naming is just for making it well plotted like Spark Whole Stage Codegen).
+
+# Spill (Experimental)
+
+Velox backend supports spilling-to-disk.
+
+Using the following configuration options to customize spilling:
+
+| Name | Default Value | Description |
+|--------------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| spark.gluten.sql.columnar.backend.velox.spillStrategy | auto | none: Disable spill on Velox backend; auto: Let Spark memory manager manage Velox's spilling |
+| spark.gluten.sql.columnar.backend.velox.spillFileSystem | local | The filesystem used to store spill data. local: The local file system. heap-over-local: Write files to JVM heap if having extra heap space. Otherwise write to local file system. |
+| spark.gluten.sql.columnar.backend.velox.aggregationSpillEnabled | true | Whether spill is enabled on aggregations |
+| spark.gluten.sql.columnar.backend.velox.joinSpillEnabled | true | Whether spill is enabled on joins |
+| spark.gluten.sql.columnar.backend.velox.orderBySpillEnabled | true | Whether spill is enabled on sorts |
+| spark.gluten.sql.columnar.backend.velox.aggregationSpillMemoryThreshold | 0 | Memory limit before spilling to disk for aggregations, per Spark task. Unit: byte |
+| spark.gluten.sql.columnar.backend.velox.joinSpillMemoryThreshold | 0 | Memory limit before spilling to disk for joins, per Spark task. Unit: byte |
+| spark.gluten.sql.columnar.backend.velox.orderBySpillMemoryThreshold | 0 | Memory limit before spilling to disk for sorts, per Spark task. Unit: byte |
+| spark.gluten.sql.columnar.backend.velox.maxSpillLevel | 4 | The max allowed spilling level with zero being the initial spilling level |
+| spark.gluten.sql.columnar.backend.velox.maxSpillFileSize | 20MB | The max allowed spill file size. If it is zero, then there is no limit |
+| spark.gluten.sql.columnar.backend.velox.minSpillRunSize | 268435456 | The min spill run size limit used to select partitions for spilling |
+| spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit | 29 | The start partition bit which is used with 'spillPartitionBits' together to calculate the spilling partition number |
+| spark.gluten.sql.columnar.backend.velox.spillPartitionBits | 2 | The number of bits used to calculate the spilling partition number. The number of spilling partitions will be power of two |
+| spark.gluten.sql.columnar.backend.velox.spillableReservationGrowthPct | 25 | The spillable memory reservation growth percentage of the previous memory reservation size |
+
+# Velox User-Defined Functions (UDF)
+
+## Introduction
+
+Velox backend supports User-Defined Functions (UDF). Users can create their own functions using the UDF interface provided in Velox backend and build libraries for these functions. At runtime, the UDF are registered at the start of applications. Once registered, Gluten will be able to parse and offload these UDF into Velox during execution.
+
+## Creating a UDF library
+
+The following steps demonstrate how to set up a UDF library project:
+
+- **Include the UDF Interface Header:** First, include the UDF interface header file [Udf.h](../../cpp/velox/udf/Udf.h) in the project file. The header file defines the `UdfEntry` struct, along with the macros for declaring the necessary functions to integrate the UDF into Gluten and Velox.
+
+- **Implement the UDF:** Implement UDF. These functions should be able to register to Velox.
+
+- **Implement the Interface Functions:** Implement the following interface functions that integrate UDF into Project Gluten:
+
+ - `getNumUdf()`: This function should return the number of UDF in the library. This is used to allocating udfEntries array as the argument for the next function `getUdfEntries`.
+
+ - `getUdfEntries(gluten::UdfEntry* udfEntries)`: This function should populate the provided udfEntries array with the details of the UDF, including function names and return types.
+
+ - `registerUdf()`: This function is called to register the UDF to Velox function registry. This is where users should register functions by calling `facebook::velox::exec::registerVecotorFunction` or other Velox APIs.
+
+ - The interface functions are mapping to marcos in [Udf.h](../../cpp/velox/udf/Udf.h). Here's an example of how to implement these functions:
+
+ ```
+ // Filename MyUDF.cpp
+
+ #include <velox/expression/VectorFunction.h>
+ #include <velox/udf/Udf.h>
+
+ const int kNumMyUdf = 1;
+
+ gluten::UdfEntry myUdf[kNumMyUdf] = {{"myudf1", "integer"}};
+
+ class MyUdf : public facebook::velox::exec::VectorFunction {
+ ... // Omit concrete implementation
+ }
+
+ static std::vector<std::shared_ptr<exec::FunctionSignature>>
+ myUdfSignatures() {
+ return {facebook::velox::exec::FunctionSignatureBuilder()
+ .returnType(myUdf[0].dataType)
+ .argumentType("integer")
+ .build()};
+ }
+
+ DEFINE_GET_NUM_UDF { return kNumMyUdf; }
+
+ DEFINE_GET_UDF_ENTRIES { udfEntries[0] = myUdf[0]; }
+
+ DEFINE_REGISTER_UDF {
+ facebook::velox::exec::registerVectorFunction(
+ myUdf[0].name, myUdfSignatures(), std::make_unique<MyUdf>());
+ }
+
+ ```
+
+## Building the UDF library
+
+To build the UDF library, users need to compile the C++ code and link to `libvelox.so`. It's recommended to create a CMakeLists.txt for the project. Here's an example:
+```
+project(myudf)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+
+set(GLUTEN_HOME /path/to/gluten)
+
+add_library(myudf SHARED "MyUDF.cpp")
+
+find_library(VELOX_LIBRARY REQUIRED NAMES velox HINTS ${GLUTEN_HOME}/cpp/build/releases NO_DEFAULT_PATH)
+
+target_include_directories(myudf PRIVATE ${GLUTEN_HOME}/cpp ${GLUTEN_HOME}/ep/build-velox/build/velox_ep)
+target_link_libraries(myudf PRIVATE ${VELOX_LIBRARY})
+```
+
+## Using UDF in Gluten
+
+Gluten loads the UDF libraries at runtime. You can upload UDF libraries via `--files` or `--archives`, and configure the libray paths using the provided Spark configuration, which accepts comma separated list of library paths.
+
+Note if running on Yarn client mode, the uploaded files are not reachable on driver side. Users should copy those files to somewhere reachable for driver and set `spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths`. This configuration is also useful when the `udfLibraryPaths` is different between driver side and executor side.
+
+- Use `--files`
+```shell
+--files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
+# Needed for Yarn client mode
+--conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/libmyudf.so
+```
+
+- Use `--archives`
+```shell
+--archives /path/to/udf_archives.zip#udf_archives
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=udf_archives
+# Needed for Yarn client mode
+--conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/udf_archives.zip
+```
+
+- Specify URI
+
+You can also specify the local or HDFS URIs to the UDF libraries or archives. Local URIs should exist on driver and every worker nodes.
+```shell
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=hdfs://path/to/library_or_archive
+```
+
+## Try the example
+
+We provided an Velox UDF example file [MyUDF.cpp](../../cpp/velox/udf/examples/MyUDF.cpp). After building gluten cpp, you can find the example library at /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+
+Start spark-shell or spark-sql with below configuration
+```
+--files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
+--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
+```
+Run query. The functions `myudf1` and `myudf2` increment the input value by a constant of 5
+```
+select myudf1(1), myudf2(100L)
+```
+The output from spark-shell will be like
+```
++----------------+------------------+
+|udfexpression(1)|udfexpression(100)|
++----------------+------------------+
+| 6| 105|
++----------------+------------------+
+```
+
+# High-Bandwidth Memory (HBM) support
+
+Gluten supports allocating memory on HBM. This feature is optional and is disabled by default. It is implemented on top of [Memkind library](http://memkind.github.io/memkind/). You can refer to memkind's [readme](https://github.com/memkind/memkind#memkind) for more details.
+
+## Build Gluten with HBM
+
+Gluten will internally build and link to a specific version of Memkind library and [hwloc](https://github.com/open-mpi/hwloc). Other dependencies should be installed on Driver and Worker node first:
+
+```bash
+sudo apt install -y autoconf automake g++ libnuma-dev libtool numactl unzip libdaxctl-dev
+```
+
+After the set-up, you can now build Gluten with HBM. Below command is used to enable this feature
+
+```bash
+cd /path/to/gluten
+
+## The script builds two jars for spark 3.2.2 and 3.3.1.
+./dev/buildbundle-veloxbe.sh --enable_hbm=ON
+```
+
+## Configure and enable HBM in Spark Application
+
+At runtime, `MEMKIND_HBW_NODES` enviroment variable is detected for configuring HBM NUMA nodes. For the explaination to this variable, please refer to memkind's manual page. This can be set for all executors through spark conf, e.g. `--conf spark.executorEnv.MEMKIND_HBW_NODES=8-15`. Note that memory allocation fallback is also supported and cannot be turned off. If HBM is unavailable or fills up, the allocator will use default(DDR) memory.
+
+# Intel® QuickAssist Technology (QAT) support
+
+Gluten supports using Intel® QuickAssist Technology (QAT) for data compression during Spark Shuffle. It benefits from QAT Hardware-based acceleration on compression/decompression, and uses Gzip as compression format for higher compression ratio to reduce the pressure on disks and network transmission.
+
+This feature is based on QAT driver library and [QATzip](https://github.com/intel/QATzip) library. Please manually download QAT driver for your system, and follow its README to build and install on all Driver and Worker node: [Intel® QuickAssist Technology Driver for Linux* – HW Version 2.0](https://www.intel.com/content/www/us/en/download/765501/intel-quickassist-technology-driver-for-linux-hw-version-2-0.html?wapkw=quickassist).
+
+## Software Requirements
+- Download QAT driver for your system, and follow its README to build and install on all Driver and Worker nodes: [Intel® QuickAssist Technology Driver for Linux* – HW Version 2.0](https://www.intel.com/content/www/us/en/download/765501/intel-quickassist-technology-driver-for-linux-hw-version-2-0.html?wapkw=quickassist).
+- Below compression libraries need to be installed on all Driver and Worker nodes:
+ - Zlib* library of version 1.2.7 or higher
+ - ZSTD* library of version 1.5.4 or higher
+ - LZ4* library
+
+## Build Gluten with QAT
+
+1. Setup ICP_ROOT environment variable to the directory where QAT driver is extracted.
+This environment variable is required during building Gluten and running Spark applications.
+It's recommended to put it in .bashrc on Driver and Worker nodes.
+
+```bash
+echo "export ICP_ROOT=/path/to/QAT_driver" >> ~/.bashrc
+source ~/.bashrc
+
+# Also set for root if running as non-root user
+sudo su -
+echo "export ICP_ROOT=/path/to/QAT_driver" >> ~/.bashrc
+exit
+```
+
+2. **This step is required if your application is running as Non-root user**.
+The users must be added to the 'qat' group after QAT drvier is installed.
+And change the amount of max locked memory for the username that is included in the group name. This can be done by specifying the limit in /etc/security/limits.conf.
+
+```bash
+sudo su -
+usermod -aG qat username # need relogin to take effect
+
+# To set 500MB add a line like this in /etc/security/limits.conf
+echo "@qat - memlock 500000" >> /etc/security/limits.conf
+
+exit
+```
+
+3. Enable huge page. This step is required to execute each time after system reboot. We recommend using systemctl to manage at system startup.
+You change the values for "max_huge_pages" and "max_huge_pages_per_process" to make sure there are enough resources for your workload.
+As for Spark applications, one process matches one executor. Within the executor, every task is allocated a maximum of 5 huge pages.
+
+```bash
+sudo su -
+
+cat << EOF > /usr/local/bin/qat_startup.sh
+#!/bin/bash
+echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+rmmod usdm_drv
+insmod $ICP_ROOT/build/usdm_drv.ko max_huge_pages=1024 max_huge_pages_per_process=32
+EOF
+
+chmod +x /usr/local/bin/qat_startup.sh
+
+cat << EOF > /etc/systemd/system/qat_startup.service
+[Unit]
+Description=Configure QAT
+
+[Service]
+ExecStart=/usr/local/bin/qat_startup.sh
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+systemctl enable qat_startup.service
+systemctl start qat_startup.service # setup immediately
+systemctl status qat_startup.service
+
+exit
+```
+
+4. After the setup, you are now ready to build Gluten with QAT. Use the command below to enable this feature:
+
+```bash
+cd /path/to/gluten
+
+## The script builds two jars for spark 3.2.2 and 3.3.1.
+./dev/buildbundle-veloxbe.sh --enable_qat=ON
+```
+
+## Enable QAT with Gzip/Zstd for shuffle compression
+
+1. To offload shuffle compression into QAT, first make sure you have the right QAT configuration file at /etc/4xxx_devX.conf. We provide a [example configuration file](../qat/4x16.conf). This configuration sets up to 4 processes that can bind to 1 QAT, and each process can use up to 16 QAT DC instances.
+
+```bash
+## run as root
+## Overwrite QAT configuration file.
+cd /etc
+for i in {0..7}; do echo "4xxx_dev$i.conf"; done | xargs -i cp -f /path/to/gluten/docs/qat/4x16.conf {}
+## Restart QAT after updating configuration files.
+adf_ctl restart
+```
+
+2. Check QAT status and make sure the status is up
+
+```bash
+adf_ctl status
+```
+
+The output should be like:
+
+```
+Checking status of all devices.
+There is 8 QAT acceleration device(s) in the system:
+ qat_dev0 - type: 4xxx, inst_id: 0, node_id: 0, bsf: 0000:6b:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev1 - type: 4xxx, inst_id: 1, node_id: 1, bsf: 0000:70:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev2 - type: 4xxx, inst_id: 2, node_id: 2, bsf: 0000:75:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev3 - type: 4xxx, inst_id: 3, node_id: 3, bsf: 0000:7a:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev4 - type: 4xxx, inst_id: 4, node_id: 4, bsf: 0000:e8:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev5 - type: 4xxx, inst_id: 5, node_id: 5, bsf: 0000:ed:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev6 - type: 4xxx, inst_id: 6, node_id: 6, bsf: 0000:f2:00.0, #accel: 1 #engines: 9 state: up
+ qat_dev7 - type: 4xxx, inst_id: 7, node_id: 7, bsf: 0000:f7:00.0, #accel: 1 #engines: 9 state: up
+```
+
+3. Extra Gluten configurations are required when starting Spark application
+
+```
+--conf spark.gluten.sql.columnar.shuffle.codec=gzip # Valid options are gzip and zstd
+--conf spark.gluten.sql.columnar.shuffle.codecBackend=qat
+```
+
+4. You can use below command to check whether QAT is working normally at run-time. The value of fw_counters should continue to increase during shuffle.
+
+```
+while :; do cat /sys/kernel/debug/qat_4xxx_0000:6b:00.0/fw_counters; sleep 1; done
+```
+
+## QAT driver references
+
+**Documentation**
+
+[README Text Files (README_QAT20.L.1.0.0-00021.txt)](https://downloadmirror.intel.com/765523/README_QAT20.L.1.0.0-00021.txt)
+
+**Release Notes**
+
+Check out the [Intel® QuickAssist Technology Software for Linux*](https://www.intel.com/content/www/us/en/content-details/632507/intel-quickassist-technology-intel-qat-software-for-linux-release-notes-hardware-version-2-0.html) - Release Notes for the latest changes in this release.
+
+**Getting Started Guide**
+
+Check out the [Intel® QuickAssist Technology Software for Linux*](https://www.intel.com/content/www/us/en/content-details/632506/intel-quickassist-technology-intel-qat-software-for-linux-getting-started-guide-hardware-version-2-0.html) - Getting Started Guide for detailed installation instructions.
+
+**Programmer's Guide**
+
+Check out the [Intel® QuickAssist Technology Software for Linux*](https://www.intel.com/content/www/us/en/content-details/743912/intel-quickassist-technology-intel-qat-software-for-linux-programmers-guide-hardware-version-2-0.html) - Programmer's Guide for software usage guidelines.
+
+For more Intel® QuickAssist Technology resources go to [Intel® QuickAssist Technology (Intel® QAT)](https://developer.intel.com/quickassist)
+
+# Intel® In-memory Analytics Accelerator (IAA/IAX) support
+
+Similar to Intel® QAT, Gluten supports using Intel® In-memory Analytics Accelerator (IAA, also called IAX) for data compression during Spark Shuffle. It benefits from IAA Hardware-based acceleration on compression/decompression, and uses Gzip as compression format for higher compression ratio to reduce the pressure on disks and network transmission.
+
+This feature is based on Intel® [QPL](https://github.com/intel/qpl).
+
+## Build Gluten with IAA
+
+Gluten will internally build and link to a specific version of QPL library, but extra environment setup is still required. Please refer to [QPL Installation Guide](https://intel.github.io/qpl/documentation/get_started_docs/installation.html) to install dependencies and configure accelerators.
+
+**This step is required if your application is running as Non-root user**. Create a group for the users who have privilege to use IAA, and grant group iaa read/write access to the IAA Work-Queues.
+
+```bash
+sudo groupadd iaa
+sudo usermod -aG iaa username # need to relogin
+sudo chgrp -R iaa /dev/iax
+sudo chmod -R g+rw /dev/iax
+```
+
+After the set-up, you can now build Gluten with QAT. Below command is used to enable this feature
+
+```bash
+cd /path/to/gluten
+
+## The script builds two jars for spark 3.2.2 and 3.3.1.
+./dev/buildbundle-veloxbe.sh --enable_iaa=ON
+```
+
+## Enable IAA with Gzip Compression for shuffle compression
+
+1. To enable QAT at run-time, first make sure you have configured the IAA Work-Queues correctly, and the file permissions of /dev/iax/wqX.0 are correct.
+
+```bash
+sudo ls -l /dev/iax
+```
+
+The output should be like:
+```
+total 0
+crw-rw---- 1 root iaa 509, 0 Apr 5 18:54 wq1.0
+crw-rw---- 1 root iaa 509, 5 Apr 5 18:54 wq11.0
+crw-rw---- 1 root iaa 509, 6 Apr 5 18:54 wq13.0
+crw-rw---- 1 root iaa 509, 7 Apr 5 18:54 wq15.0
+crw-rw---- 1 root iaa 509, 1 Apr 5 18:54 wq3.0
+crw-rw---- 1 root iaa 509, 2 Apr 5 18:54 wq5.0
+crw-rw---- 1 root iaa 509, 3 Apr 5 18:54 wq7.0
+crw-rw---- 1 root iaa 509, 4 Apr 5 18:54 wq9.0
+```
+
+2. Extra Gluten configurations are required when starting Spark application
+
+```
+--conf spark.gluten.sql.columnar.shuffle.codec=gzip
+--conf spark.gluten.sql.columnar.shuffle.codecBackend=iaa
+```
+
+## IAA references
+
+**Intel® IAA Enabling Guide**
+
+Check out the [Intel® In-Memory Analytics Accelerator (Intel® IAA) Enabling Guide](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-iaa-enabling-guide.html)
+
+**Intel® QPL Documentation**
+
+Check out the [Intel® Query Processing Library (Intel® QPL) Documentation](https://intel.github.io/qpl/index.html)
+
+# Test TPC-H or TPC-DS on Gluten with Velox backend
+
+All TPC-H and TPC-DS queries are supported in Gluten Velox backend.
+
+## Data preparation
+
+The data generation scripts are [TPC-H dategen script](../../tools/workload/tpch/gen_data/parquet_dataset/tpch_datagen_parquet.sh) and
+[TPC-DS dategen script](../../tools/workload/tpcds/gen_data/parquet_dataset/tpcds_datagen_parquet.sh).
+
+The used TPC-H and TPC-DS queries are the original ones, and can be accessed from [TPC-DS queries](../../gluten-core/src/test/resources/tpcds-queries/tpcds.queries.original)
+and [TPC-H queries](../../gluten-core/src/test/resources/tpch-queries).
+
+Some other versions of TPC-DS queries are also provided, but are **not** recommended for testing, including:
+- the modified TPC-DS queries with "Decimal-to-Double": [TPC-DS non-decimal queries](../../gluten-core/src/test/resources/tpcds-queries/tpcds.queries.no-decimal) (outdated).
+
+## Submit the Spark SQL job
+
+Submit test script from spark-shell. You can find the scala code to [Run TPC-H](../../tools/workload/tpch/run_tpch/tpch_parquet.scala) as an example. Please remember to modify
+the location of TPC-H files as well as TPC-H queries before you run the testing.
+
+```
+var parquet_file_path = "/PATH/TO/TPCH_PARQUET_PATH"
+var gluten_root = "/PATH/TO/GLUTEN"
+```
+
+Below script shows an example about how to run the testing, you should modify the parameters such as executor cores, memory, offHeap size based on your environment.
+
+```bash
+export GLUTEN_JAR = /PATH/TO/GLUTEN/package/target/<gluten-jar>
+cat tpch_parquet.scala | spark-shell --name tpch_powertest_velox \
+ --master yarn --deploy-mode client \
+ --conf spark.plugins=io.glutenproject.GlutenPlugin \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=20g \
+ --conf spark.gluten.sql.columnar.forceShuffledHashJoin=true \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --num-executors 6 \
+ --executor-cores 6 \
+ --driver-memory 20g \
+ --executor-memory 25g \
+ --conf spark.executor.memoryOverhead=5g \
+ --conf spark.driver.maxResultSize=32g
+```
+
+Refer to [Gluten configuration](../Configuration.md) for more details.
+
+## Result
+*wholestagetransformer* indicates that the offload works.
+
+
+
+## Performance
+
+Below table shows the TPC-H Q1 and Q6 Performance in a multiple-thread test (--num-executors 6 --executor-cores 6) for Velox and vanilla Spark.
+Both Parquet and ORC datasets are sf1024.
+
+| Query Performance (s) | Velox (ORC) | Vanilla Spark (Parquet) | Vanilla Spark (ORC) |
+|---------------- | ----------- | ------------- | ------------- |
+| TPC-H Q6 | 13.6 | 21.6 | 34.9 |
+| TPC-H Q1 | 26.1 | 76.7 | 84.9 |
+
+# External reference setup
+
+TO ease your first-hand experience of using Gluten, we have set up an external reference cluster. If you are interested, please contact Weiting.Chen@intel.com.
+
+# Gluten UI
+
+## Gluten event
+
+Gluten provides two events `GlutenBuildInfoEvent` and `GlutenPlanFallbackEvent`:
+
+- GlutenBuildInfoEvent, it contains the Gluten build information so that we are able to be aware of the environment when doing some debug.
+ It includes `Java Version`, `Scala Version`, `GCC Version`, `Gluten Version`, `Spark Version`, `Hadoop Version`, `Gluten Revision`, `Backend`, `Backend Revision`, etc.
+
+- GlutenPlanFallbackEvent, it contains the fallback information for each query execution.
+ Note, if the query execution is in AQE, then Gluten will post it for each stage.
+
+Developers can register `SparkListener` to handle these two Gluten events.
+
+## SQL tab
+
+Gluten provides a tab based on Spark UI, named `Gluten SQL / DataFrame`
+
+
+
+This tab contains two parts:
+
+1. The Gluten build information.
+2. SQL/Dataframe queries fallback information.
+
+If you want to disable Gluten UI, add a config when submitting `--conf spark.gluten.ui.enabled=false`.
+
+## History server
+
+Gluten UI also supports Spark history server. Add gluten-ui jar into the history server classpath, e.g., $SPARK_HOME/jars, then restart history server.
+
+## Native plan string
+
+Gluten supports inject native plan string into Spark explain with formatted mode by setting `--conf spark.gluten.sql.injectNativePlanStringToExplain=true`.
+Here is an example, how Gluten show the native plan string.
+
+```
+(9) WholeStageCodegenTransformer (2)
+Input [6]: [c1#0L, c2#1L, c3#2L, c1#3L, c2#4L, c3#5L]
+Arguments: false
+Native Plan:
+-- Project[expressions: (n3_6:BIGINT, "n0_0"), (n3_7:BIGINT, "n0_1"), (n3_8:BIGINT, "n0_2"), (n3_9:BIGINT, "n1_0"), (n3_10:BIGINT, "n1_1"), (n3_11:BIGINT, "n1_2")] -> n3_6:BIGINT, n3_7:BIGINT, n3_8:BIGINT, n3_9:BIGINT, n3_10:BIGINT, n3_11:BIGINT
+ -- HashJoin[INNER n1_1=n0_1] -> n1_0:BIGINT, n1_1:BIGINT, n1_2:BIGINT, n0_0:BIGINT, n0_1:BIGINT, n0_2:BIGINT
+ -- TableScan[table: hive_table, range filters: [(c2, Filter(IsNotNull, deterministic, null not allowed))]] -> n1_0:BIGINT, n1_1:BIGINT, n1_2:BIGINT
+ -- ValueStream[] -> n0_0:BIGINT, n0_1:BIGINT, n0_2:BIGINT
+```
+
+## Native plan with stats
+
+Gluten supports print native plan with stats to executor system output stream by setting `--conf spark.gluten.sql.debug=true`.
+Note that, the plan string with stats is task level which may cause executor log size big. Here is an example, how Gluten show the native plan string with stats.
+
+```
+I20231121 10:19:42.348845 90094332 WholeStageResultIterator.cc:220] Native Plan with stats for: [Stage: 1 TID: 16]
+-- Project[expressions: (n3_6:BIGINT, "n0_0"), (n3_7:BIGINT, "n0_1"), (n3_8:BIGINT, "n0_2"), (n3_9:BIGINT, "n1_0"), (n3_10:BIGINT, "n1_1"), (n3_11:BIGINT, "n1_2")] -> n3_6:BIGINT, n3_7:BIGINT, n3_8:BIGINT, n3_9:BIGINT, n3_10:BIGINT, n3_11:BIGINT
+ Output: 27 rows (3.56KB, 3 batches), Cpu time: 10.58us, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+ queuedWallNanos sum: 2.00us, count: 1, min: 2.00us, max: 2.00us
+ runningAddInputWallNanos sum: 626ns, count: 1, min: 626ns, max: 626ns
+ runningFinishWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ runningGetOutputWallNanos sum: 5.54us, count: 1, min: 5.54us, max: 5.54us
+ -- HashJoin[INNER n1_1=n0_1] -> n1_0:BIGINT, n1_1:BIGINT, n1_2:BIGINT, n0_0:BIGINT, n0_1:BIGINT, n0_2:BIGINT
+ Output: 27 rows (3.56KB, 3 batches), Cpu time: 223.00us, Blocked wall time: 0ns, Peak memory: 93.12KB, Memory allocations: 15
+ HashBuild: Input: 10 rows (960B, 10 batches), Output: 0 rows (0B, 0 batches), Cpu time: 185.67us, Blocked wall time: 0ns, Peak memory: 68.00KB, Memory allocations: 2, Threads: 1
+ distinctKey0 sum: 4, count: 1, min: 4, max: 4
+ hashtable.capacity sum: 4, count: 1, min: 4, max: 4
+ hashtable.numDistinct sum: 10, count: 1, min: 10, max: 10
+ hashtable.numRehashes sum: 1, count: 1, min: 1, max: 1
+ queuedWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ rangeKey0 sum: 4, count: 1, min: 4, max: 4
+ runningAddInputWallNanos sum: 1.27ms, count: 1, min: 1.27ms, max: 1.27ms
+ runningFinishWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ runningGetOutputWallNanos sum: 1.29us, count: 1, min: 1.29us, max: 1.29us
+ H23/11/21 10:19:42 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 13) in 335 ms on 10.221.97.35 (executor driver) (1/10)
+ashProbe: Input: 9 rows (864B, 3 batches), Output: 27 rows (3.56KB, 3 batches), Cpu time: 37.33us, Blocked wall time: 0ns, Peak memory: 25.12KB, Memory allocations: 13, Threads: 1
+ dynamicFiltersProduced sum: 1, count: 1, min: 1, max: 1
+ queuedWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ runningAddInputWallNanos sum: 4.54us, count: 1, min: 4.54us, max: 4.54us
+ runningFinishWallNanos sum: 83ns, count: 1, min: 83ns, max: 83ns
+ runningGetOutputWallNanos sum: 29.08us, count: 1, min: 29.08us, max: 29.08us
+ -- TableScan[table: hive_table, range filters: [(c2, Filter(IsNotNull, deterministic, null not allowed))]] -> n1_0:BIGINT, n1_1:BIGINT, n1_2:BIGINT
+ Input: 9 rows (864B, 3 batches), Output: 9 rows (864B, 3 batches), Cpu time: 630.75us, Blocked wall time: 0ns, Peak memory: 2.44KB, Memory allocations: 63, Threads: 1, Splits: 3
+ dataSourceWallNanos sum: 102.00us, count: 1, min: 102.00us, max: 102.00us
+ dynamicFiltersAccepted sum: 1, count: 1, min: 1, max: 1
+ flattenStringDictionaryValues sum: 0, count: 1, min: 0, max: 0
+ ioWaitNanos sum: 312.00us, count: 1, min: 312.00us, max: 312.00us
+ localReadBytes sum: 0B, count: 1, min: 0B, max: 0B
+ numLocalRead sum: 0, count: 1, min: 0, max: 0
+ numPrefetch sum: 0, count: 1, min: 0, max: 0
+ numRamRead sum: 0, count: 1, min: 0, max: 0
+ numStorageRead sum: 6, count: 1, min: 6, max: 6
+ overreadBytes sum: 0B, count: 1, min: 0B, max: 0B
+ prefetchBytes sum: 0B, count: 1, min: 0B, max: 0B
+ queryThreadIoLatency sum: 12, count: 1, min: 12, max: 12
+ ramReadBytes sum: 0B, count: 1, min: 0B, max: 0B
+ runningAddInputWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ runningFinishWallNanos sum: 125ns, count: 1, min: 125ns, max: 125ns
+ runningGetOutputWallNanos sum: 1.07ms, count: 1, min: 1.07ms, max: 1.07ms
+ skippedSplitBytes sum: 0B, count: 1, min: 0B, max: 0B
+ skippedSplits sum: 0, count: 1, min: 0, max: 0
+ skippedStrides sum: 0, count: 1, min: 0, max: 0
+ storageReadBytes sum: 3.44KB, count: 1, min: 3.44KB, max: 3.44KB
+ totalScanTime sum: 0ns, count: 1, min: 0ns, max: 0ns
+ -- ValueStream[] -> n0_0:BIGINT, n0_1:BIGINT, n0_2:BIGINT
+ Input: 0 rows (0B, 0 batches), Output: 10 rows (960B, 10 batches), Cpu time: 1.03ms, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
+ runningAddInputWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
+ runningFinishWallNanos sum: 54.62us, count: 1, min: 54.62us, max: 54.62us
+ runningGetOutputWallNanos sum: 1.10ms, count: 1, min: 1.10ms, max: 1.10ms
+```
+
+# Gluten Implicits
+
+Gluten provides a helper class to get the fallback summary from a Spark Dataset.
+
+```
+import org.apache.spark.sql.execution.GlutenImplicits._
+val df = spark.sql("SELECT * FROM t")
+df.fallbackSummary
+```
+
+Note that, if AQE is enabled, but the query is not materialized, then it will re-plan
+the query execution with disabled AQE. It is a workaround to get the final plan, and it may
+cause the inconsistent results with a materialized query. However, we have no choice.
diff --git a/archives/v1.1.1/docs/Velox.md b/archives/v1.1.1/docs/Velox.md
new file mode 100644
index 0000000..9d1099b
--- /dev/null
+++ b/archives/v1.1.1/docs/Velox.md
@@ -0,0 +1,10 @@
+---
+layout: page
+title: v1.1.1/Velox Backend
+nav_order: 1
+has_children: true
+parent: v1.1.1/Documentation
+grand_parent: v1.1.1
+permalink: /archives/v1.1.1/docs/velox/
+---
+# Gluten Velox Backend Documents
diff --git a/archives/v1.1.1/docs/VeloxBuildGuide.md b/archives/v1.1.1/docs/VeloxBuildGuide.md
new file mode 100644
index 0000000..2409c11
--- /dev/null
+++ b/archives/v1.1.1/docs/VeloxBuildGuide.md
@@ -0,0 +1,78 @@
+---
+layout: page
+title: v1.1.1/Build Parameters for Velox Backend
+nav_order: 2
+parent: v1.1.1/Velox Backend
+grand_parent: v1.1.1/Documentation
+permalink: /archives/v1.1.1/docs/velox/build_parameters/
+---
+## Build Parameters
+### Native build parameters for buildbundle-veloxbe.sh or builddeps-veloxbe.sh
+Please set them via `--`, e.g. `--build_type=Release`.
+
+| Parameters | Description | Default |
+|------------------------|----------------------------------------------------------------------------|---------|
+| build_type | Build type for Velox & gluten cpp, CMAKE_BUILD_TYPE. | Release |
+| build_tests | Build gluten cpp tests. | OFF |
+| build_examples | Build udf example. | OFF |
+| build_benchmarks | Build gluten cpp benchmarks. | OFF |
+| build_jemalloc | Build with jemalloc. | ON |
+| build_protobuf | Build protobuf lib. | ON |
+| enable_qat | Enable QAT for shuffle data de/compression. | OFF |
+| enable_iaa | Enable IAA for shuffle data de/compression. | OFF |
+| enable_hbm | Enable HBM allocator. | OFF |
+| enable_s3 | Build with S3 support. | OFF |
+| enable_gcs | Build with GCs support. | OFF |
+| enable_hdfs | Build with HDFS support. | OFF |
+| enable_abfs | Build with ABFS support. | OFF |
+| enable_ep_cache | Enable caching for external project build (Velox). | OFF |
+| enable_vcpkg | Enable vcpkg for static build. | OFF |
+| run_setup_script | Run setup script to install Velox dependencies. | ON |
+| velox_repo | Specify your own Velox repo to build. | "" |
+| velox_branch | Specify your own Velox branch to build. | "" |
+| velox_home | Specify your own Velox source path to build. | "" |
+| build_velox_tests | Build Velox tests. | OFF |
+| build_velox_benchmarks | Build Velox benchmarks (velox_tests and connectors will be disabled if ON) | OFF |
+| compile_arrow_java | Compile arrow java for gluten build to use to fix invalid pointer issues. | OFF |
+
+### Velox build parameters for build_velox.sh
+Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
+
+| Parameters | Description | Default |
+|--------------------|-------------------------------------------------------------------------|------------------------------------------|
+| velox_home | Specify Velox source path to build. | GLUTEN_SRC/ep/build-velox/build/velox_ep |
+| build_type | Velox build type, i.e., CMAKE_BUILD_TYPE. | Release |
+| enable_s3 | Build Velox with S3 support. | OFF |
+| enable_gcs | Build Velox with GCS support. | OFF |
+| enable_hdfs | Build Velox with HDFS support. | OFF |
+| enable_abfs | Build Velox with ABFS support. | OFF |
+| run_setup_script | Run setup script to install Velox dependencies before build. | ON |
+| enable_ep_cache | Enable and reuse cache of Velox build. | OFF |
+| build_test_utils | Build Velox with cmake arg -DVELOX_BUILD_TEST_UTILS=ON if ON. | OFF |
+| build_tests | Build Velox test. | OFF |
+| build_benchmarks | Build Velox benchmarks. | OFF |
+| compile_arrow_java | Build arrow java for gluten build to use to fix invalid pointer issues. | OFF |
+
+### Maven build parameters
+The below parameters can be set via `-P` for mvn.
+
+| Parameters | Description | Default state |
+|---------------------|------------------------------------------------------------------------------|---------------|
+| backends-velox | Build Gluten Velox backend. | disabled |
+| backends-clickhouse | Build Gluten ClickHouse backend. | disabled |
+| rss | Build Gluten with Remote Shuffle Service, only applicable for Velox backend. | disabled |
+| delta | Build Gluten with Delta Lake support. | disabled |
+| iceberg | Build Gluten with Iceberg support. | disabled |
+| spark-3.2 | Build Gluten for Spark 3.2. | enabled |
+| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
+| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
+
+## Gluten Jar for Deployment
+The gluten jar built out is under `GLUTEN_SRC/package/target/`.
+It's name pattern is `gluten-<backend_type>-bundle-spark<spark.bundle.version>_<scala.binary.version>-<os.detected.release>_<os.detected.release.version>-<project.version>.jar`.
+
+| Spark Version | spark.bundle.version | scala.binary.version |
+|---------------|----------------------|----------------------|
+| 3.2.2 | 3.2 | 2.12 |
+| 3.3.1 | 3.3 | 2.12 |
+| 3.4.2 | 3.4 | 2.12 |
diff --git a/archives/v1.1.1/docs/VeloxGCS.md b/archives/v1.1.1/docs/VeloxGCS.md
new file mode 100644
index 0000000..5e58443
--- /dev/null
+++ b/archives/v1.1.1/docs/VeloxGCS.md
@@ -0,0 +1,41 @@
+---
+layout: page
+title: v1.1.1/Using GCS with Gluten
+nav_order: 4
+parent: v1.1.1/Velox Backend
+grand_parent: v1.1.1/Documentation
+permalink: /archives/v1.1.1/docs/velox/gcs/
+---
+Object stores offered by CSPs such as GCS are important for users of Gluten to store their data. This doc will discuss all details of configs, and use cases around using Gluten with object stores. In order to use a GCS endpoint as your data source, please ensure you are using the following GCS configs in your spark-defaults.conf. If you're experiencing any issues authenticating to GCS with additional auth mechanisms, please reach out to us using the 'Issues' tab.
+
+# Working with GCS
+
+## Installing the gcloud CLI
+
+To access GCS Objects using Gluten and Velox, first you have to [download an install the gcloud CLI] (https://cloud.google.com/sdk/docs/install).
+
+
+## Configuring GCS using a user account
+
+This is recommended for regular users, follow the [instructions to authorize a user account](https://cloud.google.com/sdk/docs/authorizing#user-account).
+After these steps, no specific configuration is required for Gluten, since the authorization was handled entirely by the gcloud tool.
+
+
+## Configuring GCS using a credential file
+
+For workloads that need to be fully automated, manually authorizing can be problematic. For such cases it is better to use a json file with the credentials.
+This is described in the [instructions to configure a service account]https://cloud.google.com/sdk/docs/authorizing#service-account.
+
+Such json file with the credetials can be passed to Gluten:
+
+```sh
+spark.hadoop.fs.gs.auth.type SERVICE_ACCOUNT_JSON_KEYFILE
+spark.hadoop.fs.gs.auth.service.account.json.keyfile // path to the json file with the credentials.
+```
+
+## Configuring GCS endpoints
+
+For cases when a GCS mock is used, an optional endpoint can be provided:
+```sh
+spark.hadoop.fs.gs.storage.root.url // url to the mock gcs service including starting with http or https
+```
diff --git a/archives/v1.1.1/docs/VeloxS3.md b/archives/v1.1.1/docs/VeloxS3.md
new file mode 100644
index 0000000..541cce1
--- /dev/null
+++ b/archives/v1.1.1/docs/VeloxS3.md
@@ -0,0 +1,75 @@
+---
+layout: page
+title: v1.1.1/Using S3 with Gluten
+nav_order: 3
+parent: v1.1.1/Velox Backend
+grand_parent: v1.1.1/Documentation
+permalink: /archives/v1.1.1/docs/velox/s3/
+---
+Object stores offered by CSPs such as AWS S3 are important for users of Gluten to store their data. This doc will discuss all details of configs, and use cases around using Gluten with object stores. In order to use an S3 endpoint as your data source, please ensure you are using the following S3 configs in your spark-defaults.conf. If you're experiencing any issues authenticating to S3 with additional auth mechanisms, please reach out to us using the 'Issues' tab.
+
+# Working with S3
+
+## Configuring S3 endpoint
+
+S3 provides the endpoint based method to access the files, here's the example configuration. Users may need to modify some values based on real setup.
+
+```sh
+spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
+spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
+spark.hadoop.fs.s3a.access.key XXXXXXXXX
+spark.hadoop.fs.s3a.secret.key XXXXXXXXX
+spark.hadoop.fs.s3a.endpoint https://s3.us-west-1.amazonaws.com
+spark.hadoop.fs.s3a.connection.ssl.enabled true
+spark.hadoop.fs.s3a.path.style.access false
+```
+
+## Configuring S3 instance credentials
+
+S3 also provides other methods for accessing, you can also use instance credentials by setting the following config
+
+```
+spark.hadoop.fs.s3a.use.instance.credentials true
+```
+Note that in this case, "spark.hadoop.fs.s3a.endpoint" won't take affect as Gluten will use the endpoint set during instance creation.
+
+## Configuring S3 IAM roles
+You can also use iam role credentials by setting the following configurations. Instance credentials have higher priority than iam credentials.
+
+```
+spark.hadoop.fs.s3a.iam.role xxxx
+spark.hadoop.fs.s3a.iam.role.session.name xxxx
+```
+
+Note that `spark.hadoop.fs.s3a.iam.role.session.name` is optional.
+
+## Other authentatication methods are not supported yet
+
+- [AWS temporary credential](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html)
+
+## Log granularity of AWS C++ SDK in velox
+
+You can change log granularity of AWS C++ SDK by setting the `spark.gluten.velox.awsSdkLogLevel` configuration. The Allowed values are:
+* OFF
+* FATAL
+* ERROR
+* WARN
+* INFO
+* DEBUG
+* TRACE
+
+# Local Caching support
+
+Velox supports a local cache when reading data from HDFS/S3. The feature is very useful if remote storage is slow, e.g., reading from a public S3 bucket and stronger performance is desired. With this feature, Velox can asynchronously cache the data on local disk when reading from remote storage, and the future reading requests on already cached blocks will be serviced from local cache files. To enable the local caching feature, below configurations are required:
+
+```
+spark.gluten.sql.columnar.backend.velox.cacheEnabled // enable or disable velox cache, default false.
+spark.gluten.sql.columnar.backend.velox.memCacheSize // the total size of in-mem cache, default is 128MB.
+spark.gluten.sql.columnar.backend.velox.ssdCachePath // the folder to store the cache files, default is "/tmp".
+spark.gluten.sql.columnar.backend.velox.ssdCacheSize // the total size of the SSD cache, default is 128MB. Velox will do in-mem cache only if this value is 0.
+spark.gluten.sql.columnar.backend.velox.ssdCacheShards // the shards of the SSD cache, default is 1.
+spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads // the IO threads for cache promoting, default is 1. Velox will try to do "read-ahead" if this value is bigger than 1
+spark.gluten.sql.columnar.backend.velox.ssdODirect // enable or disable O_DIRECT on cache write, default false.
+```
+
+It's recommended to mount SSDs to the cache path to get the best performance of local caching. On the start up of Spark context, the cache files will be allocated under "spark.gluten.sql.columnar.backend.velox.cachePath", with UUID based suffix, e.g. "/tmp/cache.13e8ab65-3af4-46ac-8d28-ff99b2a9ec9b0". Gluten is not able to reuse older caches for now, and the old cache files are left there after Spark context shutdown.
diff --git a/archives/v1.1.1/docs/index.md b/archives/v1.1.1/docs/index.md
new file mode 100644
index 0000000..661ef75
--- /dev/null
+++ b/archives/v1.1.1/docs/index.md
@@ -0,0 +1,12 @@
+---
+layout: page
+title: v1.1.1/Documentation
+nav_order: 2
+has_children: true
+parent: v1.1.1
+grand_parent: Archives
+permalink: /archives/v1.1.1/docs/
+---
+# Gluten Documents by version
+
+
diff --git a/archives/v1.1.1/images/apache-incubator.svg b/archives/v1.1.1/images/apache-incubator.svg
new file mode 100644
index 0000000..dc152b4
--- /dev/null
+++ b/archives/v1.1.1/images/apache-incubator.svg
@@ -0,0 +1 @@
+{"payload":{"allShortcutsEnabled":false,"fileTree":{"static/img":{"items":[{"name":"benchmarks","path":"static/img/benchmarks","contentType":"directory"},{"name":"apache-incubator.svg","path":"static/img/apache-incubator.svg","contentType":"file"},{"name":"case1.png","path":"static/img/case1.png","contentType":"file"},{"name":"case2.png","path":"static/img/case2.png","contentType":"file"},{"name":"favicon.ico","path":"static/img/favicon.ico","contentType":"file"},{"name":"fury_banner.png [...]
\ No newline at end of file
diff --git a/archives/v1.1.1/images/gluten-logo-blue.png b/archives/v1.1.1/images/gluten-logo-blue.png
new file mode 100644
index 0000000..801fd9b
Binary files /dev/null and b/archives/v1.1.1/images/gluten-logo-blue.png differ
diff --git a/archives/v1.1.1/index.md b/archives/v1.1.1/index.md
new file mode 100644
index 0000000..bb68ad6
--- /dev/null
+++ b/archives/v1.1.1/index.md
@@ -0,0 +1,11 @@
+---
+layout: page
+title: v1.1.1
+nav_order: 1
+has_children: true
+parent: Archives
+permalink: /archives/v1.1.1/
+---
+# Gluten Documents by version
+
+
diff --git a/contact-us.md b/contact-us.md
index ba3230f..19160ed 100644
--- a/contact-us.md
+++ b/contact-us.md
@@ -1,7 +1,7 @@
---
layout: page
title: Contact Us
-nav_order: 6
+nav_order: 7
---
# Contact Us
diff --git a/contributing.md b/contributing.md
index 5444ed1..1ade03c 100644
--- a/contributing.md
+++ b/contributing.md
@@ -1,7 +1,7 @@
---
layout: page
title: Contributing to Gluten
-nav_order: 5
+nav_order: 6
---
## How to become a committer
diff --git a/release.md b/release.md
index 71fc1f8..af8f01b 100644
--- a/release.md
+++ b/release.md
@@ -1,7 +1,7 @@
---
layout: page
title: Gluten Release
-nav_order: 4
+nav_order: 5
---
[Gluten](https://github.com/apache/incubator-gluten) is a plugin for Apache Spark to double SparkSQL's performance.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org