You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2026/05/21 09:01:32 UTC
(paimon) 01/01: [docs] Migrate documentation from Hugo to Docusaurus
This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch refactor_docs
in repository https://gitbox.apache.org/repos/asf/paimon.git
commit b2007c4cac6674bcdf50b59f6e08004598bbd57e
Author: JingsongLi <ji...@gmail.com>
AuthorDate: Thu May 21 16:30:23 2026 +0800
[docs] Migrate documentation from Hugo to Docusaurus
- Replace Hugo (hugo-book theme) with Docusaurus 3
- Convert all 145 markdown files (shortcodes, tabs, hints, refs)
- Add custom components (Stable, Unstable, ConfigTable, Label)
- Add remark plugin for @@VERSION@@ variable interpolation
- Add local search plugin (@easyops-cn/docusaurus-search-local)
- Card-style homepage with navigation to all sections
- CI: use yarn to avoid npm crash on GitHub Actions
- Update RAT exclusions for Docusaurus file structure
Co-Authored-By: Claude Opus 4.6 <no...@anthropic.com>
---
.github/workflows/docs-tests.yml | 28 +-
docs/.gitignore | 15 +-
docs/README.md | 140 +-
docs/assets/_custom.scss | 239 -
docs/assets/_fonts.scss | 25 -
docs/assets/github.css | 87 -
docs/assets/search-data.js | 50 -
docs/babel.config.js | 3 +
docs/config.toml | 85 -
docs/content/_index.md | 89 -
docs/content/append-table/_index.md | 25 -
docs/content/cdc-ingestion/_index.md | 25 -
docs/content/concepts/_index.md | 25 -
docs/content/ecosystem/_index.md | 25 -
docs/content/flink/_index.md | 25 -
docs/content/iceberg/_index.md | 25 -
docs/content/learn-paimon/_index.md | 25 -
docs/content/maintenance/_index.md | 25 -
docs/content/maintenance/configurations.md | 91 -
docs/content/migration/_index.md | 25 -
docs/content/program-api/_index.md | 25 -
docs/content/project/_index.md | 26 -
docs/content/project/download.md | 150 -
docs/content/pypaimon/_index.md | 25 -
docs/content/spark/_index.md | 25 -
.../blob.md => docs/append-table/blob.mdx} | 75 +-
.../bucketed.md => docs/append-table/bucketed.mdx} | 36 +-
.../append-table/data-evolution.md | 10 +-
.../append-table/global-index.mdx} | 51 +-
.../append-table/incremental-clustering.mdx} | 67 +-
.../rest/_index.md => docs/append-table/index.md} | 6 +-
.../overview.md => docs/append-table/overview.mdx} | 39 +-
.../{content => docs}/append-table/row-tracking.md | 6 +-
docs/{content => docs}/append-table/vector.md | 6 +-
.../cdc-ingestion/flink-cdc.mdx} | 24 +-
.../spec/_index.md => docs/cdc-ingestion/index.md} | 6 +-
.../cdc-ingestion/kafka-cdc.mdx} | 136 +-
docs/{content => docs}/cdc-ingestion/mongo-cdc.md | 69 +-
docs/{content => docs}/cdc-ingestion/mysql-cdc.md | 68 +-
.../cdc-ingestion/overview.mdx} | 24 +-
.../cdc-ingestion/postgres-cdc.md | 35 +-
docs/{content => docs}/cdc-ingestion/pulsar-cdc.md | 50 +-
docs/{content => docs}/concepts/basic-concepts.md | 12 +-
docs/{content => docs}/concepts/catalog.md | 8 +-
.../concepts/concurrency-control.md | 12 +-
docs/{content => docs}/concepts/data-types.md | 8 +-
docs/{content => docs}/concepts/functions.md | 8 +-
.../rest/_index.md => docs/concepts/index.md} | 6 +-
docs/{content => docs}/concepts/overview.md | 8 +-
docs/{content => docs}/concepts/rest/bear.md | 6 +-
docs/{content => docs}/concepts/rest/dlf.md | 12 +-
.../rest/_index.md => docs/concepts/rest/index.md} | 6 +-
docs/{content => docs}/concepts/rest/overview.md | 18 +-
docs/{content => docs}/concepts/rest/pvfs.md | 6 +-
docs/{content => docs}/concepts/rest/rest-api.md | 7 +-
.../tables.md => docs/concepts/rest/tables.mdx} | 81 +-
docs/{content => docs}/concepts/spec/datafile.md | 6 +-
docs/{content => docs}/concepts/spec/fileformat.md | 10 +-
.../concepts/spec/fileindex.mdx} | 25 +-
.../rest/_index.md => docs/concepts/spec/index.md} | 6 +-
docs/{content => docs}/concepts/spec/manifest.md | 8 +-
docs/{content => docs}/concepts/spec/overview.md | 8 +-
docs/{content => docs}/concepts/spec/schema.md | 6 +-
docs/{content => docs}/concepts/spec/snapshot.md | 6 +-
docs/{content => docs}/concepts/spec/tableindex.md | 8 +-
.../concepts/system-tables.mdx} | 41 +-
docs/{content => docs}/concepts/views.md | 12 +-
docs/{content => docs}/ecosystem/amoro.md | 8 +-
docs/{content => docs}/ecosystem/doris.md | 6 +-
.../ecosystem/hive.md => docs/ecosystem/hive.mdx} | 67 +-
.../rest/_index.md => docs/ecosystem/index.md} | 6 +-
docs/{content => docs}/ecosystem/overview.md | 8 +-
docs/{content => docs}/ecosystem/starrocks.md | 6 +-
docs/{content => docs}/ecosystem/trino.md | 22 +-
docs/{content => docs}/flink/action-jars.md | 49 +-
.../consumer-id.md => docs/flink/consumer-id.mdx} | 47 +-
docs/{content => docs}/flink/default-value.md | 6 +-
.../rest/_index.md => docs/flink/index.md} | 6 +-
docs/{content => docs}/flink/procedures.md | 6 +-
.../quick-start.md => docs/flink/quick-start.mdx} | 87 +-
docs/{content => docs}/flink/savepoint.md | 10 +-
docs/{content => docs}/flink/sql-alter.md | 30 +-
docs/{content => docs}/flink/sql-ddl.md | 30 +-
.../sql-lookup.md => docs/flink/sql-lookup.mdx} | 29 +-
.../sql-query.md => docs/flink/sql-query.mdx} | 65 +-
.../sql-write.md => docs/flink/sql-write.mdx} | 61 +-
.../iceberg/append-table.mdx} | 57 +-
.../iceberg/configurations.mdx} | 11 +-
.../ecosystem.md => docs/iceberg/ecosystem.mdx} | 25 +-
docs/{content => docs}/iceberg/hive-catalog.md | 6 +-
docs/{content => docs}/iceberg/iceberg-tags.md | 6 +-
.../rest/_index.md => docs/iceberg/index.md} | 6 +-
docs/{content => docs}/iceberg/overview.md | 12 +-
.../iceberg/primary-key-table.mdx} | 61 +-
.../iceberg/rest-catalog.mdx} | 33 +-
docs/docs/index.mdx | 133 +
.../rest/_index.md => docs/learn-paimon/index.md} | 6 +-
.../learn-paimon/scenario-guide.mdx} | 83 +-
.../learn-paimon/understand-files.mdx} | 88 +-
docs/docs/maintenance/configurations.mdx | 102 +
.../maintenance/dedicated-compaction.mdx} | 154 +-
.../maintenance/filesystems.mdx} | 372 +-
.../rest/_index.md => docs/maintenance/index.md} | 6 +-
.../maintenance/manage-branches.mdx} | 101 +-
.../maintenance/manage-partitions.md | 14 +-
.../maintenance/manage-privileges.mdx} | 81 +-
.../maintenance/manage-snapshots.mdx} | 141 +-
.../maintenance/manage-tags.mdx} | 86 +-
docs/{content => docs}/maintenance/metrics.md | 20 +-
.../maintenance/rescale-bucket.md | 6 +-
.../maintenance/write-performance.md | 8 +-
.../{content => docs}/migration/clone-to-paimon.md | 12 +-
.../rest/_index.md => docs/migration/index.md} | 6 +-
.../migration/migration-from-hive.mdx} | 57 +-
.../migration/upsert-to-partitioned.mdx} | 61 +-
.../primary-key-table/chain-table.md | 8 +-
.../primary-key-table/changelog-producer.md | 26 +-
.../primary-key-table/compaction.md | 22 +-
.../primary-key-table/data-distribution.md | 20 +-
.../_index.md => docs/primary-key-table/index.md} | 6 +-
.../merge-engine/aggregation.mdx} | 109 +-
.../primary-key-table/merge-engine/first-row.md | 22 +-
.../primary-key-table/merge-engine/index.md} | 4 +-
.../primary-key-table/merge-engine/overview.md | 14 +-
.../merge-engine/partial-update.md | 28 +-
.../primary-key-table/overview.md | 18 +-
.../primary-key-table/pk-clustering-override.md | 12 +-
.../primary-key-table/query-performance.md | 18 +-
.../primary-key-table/sequence-rowkind.mdx} | 27 +-
.../primary-key-table/table-mode.md | 24 +-
docs/{content => docs}/program-api/catalog-api.md | 6 +-
docs/{content => docs}/program-api/cpp-api.md | 8 +-
.../program-api/file-cache.mdx} | 22 +-
.../program-api/flink-api.mdx} | 33 +-
.../rest/_index.md => docs/program-api/index.md} | 6 +-
.../java-api.md => docs/program-api/java-api.mdx} | 35 +-
.../rest-api.md => docs/program-api/rest-api.mdx} | 25 +-
docs/{content => docs}/program-api/rust-api.md | 6 +-
docs/{content => docs}/project/committer.md | 6 +-
docs/{content => docs}/project/contributing.md | 16 +-
docs/docs/project/download.mdx | 179 +
.../rest/_index.md => docs/project/index.md} | 6 +-
docs/{content => docs}/pypaimon/cli.md | 10 +-
docs/{content => docs}/pypaimon/daft.md | 6 +-
docs/{content => docs}/pypaimon/data-evolution.md | 8 +-
docs/{content => docs}/pypaimon/fuse-support.md | 6 +-
docs/{content => docs}/pypaimon/global-index.md | 10 +-
.../rest/_index.md => docs/pypaimon/index.md} | 6 +-
docs/{content => docs}/pypaimon/manage-tags.md | 8 +-
docs/{content => docs}/pypaimon/overview.md | 6 +-
.../pypaimon/pyjindosdk-support.md | 6 +-
.../python-api.md => docs/pypaimon/python-api.mdx} | 86 +-
docs/{content => docs}/pypaimon/pytorch.md | 6 +-
docs/{content => docs}/pypaimon/ray-data.md | 6 +-
docs/{content => docs}/pypaimon/sql.md | 6 +-
docs/{content => docs}/pypaimon/system-tables.md | 6 +-
docs/{content => docs}/spark/auxiliary.md | 6 +-
docs/{content => docs}/spark/dataframe.md | 6 +-
docs/{content => docs}/spark/default-value.md | 6 +-
.../rest/_index.md => docs/spark/index.md} | 6 +-
docs/{content => docs}/spark/procedures.md | 6 +-
.../quick-start.md => docs/spark/quick-start.mdx} | 127 +-
docs/{content => docs}/spark/sql-alter.md | 12 +-
docs/{content => docs}/spark/sql-ddl.md | 12 +-
docs/{content => docs}/spark/sql-functions.md | 6 +-
docs/{content => docs}/spark/sql-query.md | 12 +-
docs/{content => docs}/spark/sql-upsert.md | 6 +-
docs/{content => docs}/spark/sql-write.md | 18 +-
.../spark/structured-streaming.md | 16 +-
docs/{content/versions.md => docs/versions.mdx} | 11 +-
docs/docusaurus.config.js | 181 +
.../generated/catalog_configuration.html | 18 -
.../generated/cdc_configuration.html | 18 -
.../generated/core_configuration.html | 18 -
.../generated/flink_catalog_configuration.html | 18 -
.../generated/flink_connector_configuration.html | 18 -
.../generated/hive_catalog_configuration.html | 18 -
.../generated/hive_connector_configuration.html | 18 -
.../generated/iceberg_configuration.html | 18 -
.../generated/jdbc_catalog_configuration.html | 18 -
.../generated/kafka_sync_database.html | 18 -
.../generated/kafka_sync_table.html | 18 -
.../generated/mongodb_functions.html | 18 -
.../generated/mongodb_operator.html | 18 -
.../generated/mongodb_path_example.html | 18 -
.../generated/mongodb_sync_database.html | 18 -
.../generated/mongodb_sync_table.html | 18 -
.../generated/mysql_sync_database.html | 18 -
.../generated/mysql_sync_table.html | 18 -
.../generated/orc_configuration.html | 18 -
.../shortcodes => }/generated/other_functions.html | 18 -
.../generated/postgres_sync_table.html | 18 -
.../generated/pulsar_sync_database.html | 18 -
.../generated/pulsar_sync_table.html | 18 -
.../generated/rocksdb_configuration.html | 18 -
.../shortcodes => }/generated/sort-compact.html | 18 -
.../generated/spark_catalog_configuration.html | 18 -
.../generated/spark_connector_configuration.html | 18 -
.../generated/temporal_functions.html | 18 -
docs/layouts/_default/baseof.html | 137 -
docs/layouts/partials/docs/footer.html | 28 -
.../partials/docs/inject/content-before.html | 37 -
docs/layouts/partials/docs/inject/footer.html | 34 -
docs/layouts/partials/docs/inject/head.html | 28 -
docs/layouts/partials/docs/inject/menu-after.html | 76 -
docs/layouts/partials/docs/inject/menu-before.html | 25 -
docs/layouts/partials/docs/interpolate.html | 24 -
docs/layouts/partials/docs/menu-filetree.html | 68 -
docs/layouts/partials/docs/menu.html | 42 -
docs/layouts/partials/docs/simple-title.html | 33 -
docs/layouts/partials/docs/title.html | 42 -
docs/layouts/partials/docs/toc.html | 23 -
docs/layouts/shortcodes/all_versions.html | 30 -
docs/layouts/shortcodes/artifact.html | 64 -
docs/layouts/shortcodes/beta.html | 25 -
docs/layouts/shortcodes/center.html | 25 -
docs/layouts/shortcodes/gh_link.html | 29 -
docs/layouts/shortcodes/github_repo.html | 22 -
docs/layouts/shortcodes/img.html | 34 -
docs/layouts/shortcodes/javadoc.html | 26 -
docs/layouts/shortcodes/label.html | 29 -
docs/layouts/shortcodes/presto_github_repo.html | 22 -
docs/layouts/shortcodes/query_state_warning.html | 20 -
.../layouts/shortcodes/redoc_rest_catalog_api.html | 26 -
docs/layouts/shortcodes/scala_version.html | 22 -
docs/layouts/shortcodes/selectable.html | 30 -
docs/layouts/shortcodes/stable.html | 23 -
docs/layouts/shortcodes/tab.html | 32 -
docs/layouts/shortcodes/tabs.html | 44 -
docs/layouts/shortcodes/top.html | 24 -
docs/layouts/shortcodes/trino_github_repo.html | 22 -
docs/layouts/shortcodes/unstable.html | 23 -
docs/layouts/shortcodes/version.html | 22 -
docs/package.json | 45 +
docs/redirects.js | 507 ++
docs/scripts/fix-links-v2.js | 86 +
docs/scripts/fix-links.js | 142 +
docs/scripts/fix-mdx-angles.js | 115 +
docs/scripts/fix-mdx-braces.js | 123 +
docs/scripts/migrate.js | 494 +
docs/setup_hugo.sh | 42 -
docs/sidebars.js | 318 +
docs/src/components/AllVersions.js | 16 +
docs/src/components/ConfigTable.js | 10 +
docs/src/components/Label.js | 5 +
docs/src/components/Stable.js | 8 +
docs/src/components/Unstable.js | 8 +
docs/src/css/custom.css | 213 +
docs/src/plugins/remark-variable-interpolation.js | 49 +
docs/static/js/anchor.min.js | 6 -
docs/static/js/flink.js | 182 -
docs/static/js/redoc.standalone.js | 1829 ----
docs/themes/book | 1 -
docs/yarn.lock | 9435 ++++++++++++++++++++
paimon-docs/pom.xml | 2 +-
.../java/org/apache/paimon/docs/util/Utils.java | 4 +-
pom.xml | 17 +-
257 files changed, 14492 insertions(+), 6846 deletions(-)
diff --git a/.github/workflows/docs-tests.yml b/.github/workflows/docs-tests.yml
index 6f22200974..d6d1df1987 100644
--- a/.github/workflows/docs-tests.yml
+++ b/.github/workflows/docs-tests.yml
@@ -20,9 +20,6 @@ on:
paths:
- 'docs/**'
-env:
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }}
cancel-in-progress: true
@@ -30,20 +27,25 @@ concurrency:
jobs:
build:
runs-on: ubuntu-latest
+ timeout-minutes: 15
steps:
- name: Checkout code
- uses: actions/checkout@v6
+ uses: actions/checkout@v4
- - name: Setup Hugo
- uses: peaceiris/actions-hugo@v3
+ - name: Setup Node.js
+ uses: actions/setup-node@v4
with:
- hugo-version: '0.124.1'
- extended: true
+ node-version: '20'
- - name: Build
+ - name: Install dependencies
+ working-directory: docs
run: |
- cd docs
- # Fetch the theme submodule
- git submodule update --init --recursive
- hugo
+ npm install -g yarn
+ yarn install --frozen-lockfile || yarn install
+
+ - name: Build
+ working-directory: docs
+ env:
+ NODE_OPTIONS: --max-old-space-size=4096
+ run: yarn build
diff --git a/docs/.gitignore b/docs/.gitignore
index 6a6c3e4960..cbd7bbba73 100644
--- a/docs/.gitignore
+++ b/docs/.gitignore
@@ -1,10 +1,5 @@
-.bundle/
-.jekyll-metadata
-.jekyll-cache/
-.rubydeps/
-ruby2/.bundle/
-ruby2/.rubydeps/
-public/
-resources/
-.hugo_build.lock
-.DS_Store
+node_modules/
+build/
+.docusaurus/
+.cache-loader/
+package-lock.json
diff --git a/docs/README.md b/docs/README.md
index bb1d19b6e4..d05c37f3ba 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,100 +1,96 @@
-This README gives an overview of how to build and contribute to the
-documentation of Paimon.
+# Apache Paimon Documentation
-The documentation is included with the source of Paimon in order to ensure
-that you always have docs corresponding to your checked-out version.
+This directory contains the source for the Apache Paimon documentation site, built with [Docusaurus 3](https://docusaurus.io/).
-# Requirements
+## Prerequisites
-### Install Hugo
+- Node.js >= 18
+- Yarn (`npm install -g yarn`)
-Make sure you have installed Hugo on your system.
-Note: An extended version of Hugo <= [0.124.1](https://github.com/gohugoio/hugo/releases/tag/v0.124.1) is required.
+## Quick Start
-From this directory:
+```bash
+# Install dependencies
+yarn install
-```sh
-# Install Hugo
-./setup_hugo.sh
-
-# Fetch the theme submodule
-git submodule update --init --recursive
+# Start development server (with hot reload)
+yarn start
```
-### Start Hugo local server
-
-From this directory:
-
-```sh
-hugo -b "" serve
-```
+The site will be available at http://localhost:3000/docs/master/.
-The site can be viewed at http://localhost:1313/
+## Build
-# Contribute
+```bash
+# Production build
+yarn build
-## Markdown
+# Preview the production build locally
+yarn serve
+```
-The documentation pages are written in
-[Markdown](http://daringfireball.net/projects/markdown/syntax). It is possible
-to use [GitHub flavored
-syntax](http://github.github.com/github-flavored-markdown) and intermix plain
-html.
+## Project Structure
-## Front matter
+```
+docs/
+├── docs/ # Markdown/MDX content files
+├── generated/ # Auto-generated config tables (from paimon-docs module)
+├── src/
+│ ├── components/ # Custom React components (Stable, Unstable, ConfigTable, etc.)
+│ ├── css/ # Custom styles
+│ └── plugins/ # Remark plugins (variable interpolation)
+├── static/ # Static assets (images, logos, OpenAPI spec)
+├── scripts/ # Migration utilities
+├── docusaurus.config.js
+├── sidebars.js
+└── package.json
+```
-In addition to Markdown, every page contains a Jekyll front matter, which
-specifies the title of the page and the layout to use. The title is used as the
-top-level heading for the page.
+## Regenerating Config Tables
- ---
- title: "Title of the Page"
- ---
+The 28 HTML config tables in `generated/` are produced by the `paimon-docs` Maven module. To regenerate:
- ---
- title: "Title of the Page" <-- Title rendered in the side nav
- weight: 1 <-- Weight controls the ordering of pages in the side nav
- type: docs <-- required
- aliases: <-- Alias to setup redirect from removed page to this one
- - /alias/to/removed/page.html
- ---
+```bash
+cd .. && mvn -pl paimon-docs -am package -DskipTests
+```
-## Structure
+## Adding a New Page
-### Page
+1. Create a `.md` (or `.mdx` if you need React components like Tabs) file under `docs/`
+2. Add frontmatter:
+ ```yaml
+ ---
+ title: "Page Title"
+ sidebar_position: 5
+ ---
+ ```
+3. The page will automatically appear in the sidebar based on `sidebar_position`
-#### Headings
+## Using Tabs
-All documents are structured with headings. From these headings, you can
-automatically generate a page table of contents (see below).
+```mdx
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
-```
-# Level-1 Heading <- Used for the title of the page
-## Level-2 Heading <- Start with this one for content
-### Level-3 heading
-#### Level-4 heading
-##### Level-5 heading
-```
+<Tabs groupId="engine">
+<TabItem value="flink" label="Flink">
-Please stick to the "logical order" when using the headlines, e.g. start with
-level-2 headings and use level-3 headings for subsections, etc. Don't use a
-different ordering, because you don't like how a headline looks.
+Flink content here.
-#### Table of Contents
+</TabItem>
+<TabItem value="spark" label="Spark">
-Table of contents are added automatically to every page, based on heading levels
-2 - 4. The ToC can be omitted by adding the following to the front matter of
-the page:
+Spark content here.
- ---
- bookToc: false
- ---
+</TabItem>
+</Tabs>
+```
-### ShortCodes
+## Version Variables
-Paimon uses [shortcodes](https://gohugo.io/content-management/shortcodes/) to add
-custom functionality to its documentation markdown.
+Use `@@VERSION@@` in markdown and it will be replaced with the current version (`1.5-SNAPSHOT`) at build time. Other available tokens:
-Its implementation and documentation can be found at
-`./layouts/shortcodes/artifact.html`. Please refer to `./layouts/shortcodes/`
-for other shortcodes available.
+- `@@VERSION@@` — Paimon version
+- `@@FLINK_VERSION@@` — Flink version
+- `@@BRANCH@@` — Git branch
+- `@@GITHUB_REPO@@` — GitHub repo URL
diff --git a/docs/assets/_custom.scss b/docs/assets/_custom.scss
deleted file mode 100644
index 33ccbb4060..0000000000
--- a/docs/assets/_custom.scss
+++ /dev/null
@@ -1,239 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-@import "github";
-
-.link {
- padding-bottom: 5px;
-}
-
-.appetizer {
- color: #FBB142;
-}
-
-.maindish {
- color: #7E4F89;
-}
-
-.dessert {
- color: #E6526F;
-}
-
-.book-menu nav {
- background: #f8f8f8;
-}
-
-.book-page {
- padding: 2rem 2rem;
-}
-
-.book-search input {
- background: white;
-}
-
-.markdown a {
- text-decoration: none;
- color: #05b;
-}
-
-.markdown a:visited {
- text-decoration: none;
- color: #05b;
-}
-
-.markdown {
- line-height: 1.43;
-
- h1,
- h2,
- h3,
- h4,
- h5,
- h6 {
- font-weight: 500;
- padding-top: 0;
- margin-top: 1em;
- }
-}
-
-body {
- letter-spacing: normal;
- -webkit-font-smoothing: auto;
-}
-
-aside nav ul {
- li {
- margin: 0.5em 0;
- }
-}
-
-.book-search {
- border: 2px solid #ebebeb;
-}
-
-@media screen and (max-width: 768px) {
- .toc {
- display: none;
- }
-}
-
-aside.book-menu nav {
- a:hover {
- font-weight: bold;
- opacity: 1.0;
- }
-
- a.active {
- font-weight: bold;
- color: var(--body-font-color);
- }
-}
-
-aside.book-menu > li {
- padding: 10px 5px 5px 5px;
-}
-
-aside.book-toc {
- h3 {
- margin-top: 0;
- padding-top: 0;
- font-size: 1.2em;
- }
-}
-
-html {
- line-height: 1.43;
-}
-
-h1, h2, h3, h4, h5, h6 {
- line-height: 1.1;
-}
-
-h1, h2, h3 {
- margin-top: 20px;
- margin-bottom: 10px;
-}
-
-h2, h3, h4 {
- padding-top: 1em;
-}
-
-h1 {
- font-size: 36px;
-}
-
-h2 {
- font-size: 30px;
- border-bottom: 1px solid #e5e5e5;
-}
-
-h3 {
- font-size: 24px;
-}
-
-h4 {
- font-size: 18px;
-}
-
-.markdown code {
- background: white;
- padding: 0;
- border-radius: 0;
-}
-
-pre.chroma code {
- line-height: 1.43;
-}
-
-.book-languages {
- border: 2px solid black;
-}
-
-.menu-break {
- opacity: 0.1;
-}
-
-#book-search-results {
- padding: 2px;
- background-color: white;
-}
-
-.label {
- display: inline;
- padding: .2em .6em .3em;
- font-size: 75%;
- font-weight: 700;
- line-height: 1;
- color: #fff;
- text-align: center;
- white-space: nowrap;
- vertical-align: baseline;
- border-radius: .25em;
- background-color: #337ab7;
-}
-
-.expand-toc {
- position: fixed;
- top: 2em;
- right: 5em;
- display: none;
-}
-
-.container {
- max-width: 90rem;
-}
-
-#book-search-input:focus {
- outline: none;
-}
-
-.rest-api h5 {
- margin-top: .5em;
- margin-bottom: .5em;
- font-size: 1em;
-}
-
-.rest-api tbody {
- display: table;
- width: 100%;
- background: white;
-}
-
-.rest-api td {
- background: white;
-}
-
-.rest-api .book-expand label {
- padding: 0rem 0rem;
- background: white;
-}
-
-.rest-api .book-expand {
- background: white;
-}
-
-.rest-api .book-expand .book-expand-head {
- background: white;
-}
-
-.configuration td {
- background: white;
-}
-
-.markdown table tr:nth-child(2n) {
- background: white;
-}
\ No newline at end of file
diff --git a/docs/assets/_fonts.scss b/docs/assets/_fonts.scss
deleted file mode 100644
index dc57189cf0..0000000000
--- a/docs/assets/_fonts.scss
+++ /dev/null
@@ -1,25 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-body {
- font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
- font-size: 14px;
-}
-
-code {
- font-family: "Menlo", "Lucida Console", monospace;
-}
\ No newline at end of file
diff --git a/docs/assets/github.css b/docs/assets/github.css
deleted file mode 100644
index 25600e34e7..0000000000
--- a/docs/assets/github.css
+++ /dev/null
@@ -1,87 +0,0 @@
-/**
- * Syntax highlighting generated via
- * hugo gen chromastyles --style=github > chroma.css
- */
-
-/* Background */ .chroma { background-color: #ffffff }
-/* Other */ .chroma .x { }
-/* Error */ .chroma .err { color: #a61717; background-color: #e3d2d2 }
-/* LineTableTD */ .chroma .lntd { vertical-align: top; padding: 0; margin: 0; border: 0; }
-/* LineTable */ .chroma .lntable { border-spacing: 0; padding: 0; margin: 0; border: 0; width: auto; overflow: auto; display: block; }
-/* LineHighlight */ .chroma .hl { display: block; width: 100%;background-color: #ffffcc }
-/* LineNumbersTable */ .chroma .lnt { margin-right: 0.4em; padding: 0 0.4em 0 0.4em;color: #7f7f7f }
-/* LineNumbers */ .chroma .ln { margin-right: 0.4em; padding: 0 0.4em 0 0.4em;color: #7f7f7f }
-/* Keyword */ .chroma .k { color: #000000; font-weight: bold }
-/* KeywordConstant */ .chroma .kc { color: #000000; font-weight: bold }
-/* KeywordDeclaration */ .chroma .kd { color: #000000; font-weight: bold }
-/* KeywordNamespace */ .chroma .kn { color: #000000; font-weight: bold }
-/* KeywordPseudo */ .chroma .kp { color: #000000; font-weight: bold }
-/* KeywordReserved */ .chroma .kr { color: #000000; font-weight: bold }
-/* KeywordType */ .chroma .kt { color: #445588; font-weight: bold }
-/* Name */ .chroma .n { }
-/* NameAttribute */ .chroma .na { color: #008080 }
-/* NameBuiltin */ .chroma .nb { color: #0086b3 }
-/* NameBuiltinPseudo */ .chroma .bp { color: #999999 }
-/* NameClass */ .chroma .nc { color: #445588; font-weight: bold }
-/* NameConstant */ .chroma .no { color: #008080 }
-/* NameDecorator */ .chroma .nd { color: #3c5d5d; font-weight: bold }
-/* NameEntity */ .chroma .ni { color: #800080 }
-/* NameException */ .chroma .ne { color: #990000; font-weight: bold }
-/* NameFunction */ .chroma .nf { color: #990000; font-weight: bold }
-/* NameFunctionMagic */ .chroma .fm { }
-/* NameLabel */ .chroma .nl { color: #990000; font-weight: bold }
-/* NameNamespace */ .chroma .nn { color: #555555 }
-/* NameOther */ .chroma .nx { }
-/* NameProperty */ .chroma .py { }
-/* NameTag */ .chroma .nt { color: #000080 }
-/* NameVariable */ .chroma .nv { color: #008080 }
-/* NameVariableClass */ .chroma .vc { color: #008080 }
-/* NameVariableGlobal */ .chroma .vg { color: #008080 }
-/* NameVariableInstance */ .chroma .vi { color: #008080 }
-/* NameVariableMagic */ .chroma .vm { }
-/* Literal */ .chroma .l { }
-/* LiteralDate */ .chroma .ld { }
-/* LiteralString */ .chroma .s { color: #dd1144 }
-/* LiteralStringAffix */ .chroma .sa { color: #dd1144 }
-/* LiteralStringBacktick */ .chroma .sb { color: #dd1144 }
-/* LiteralStringChar */ .chroma .sc { color: #dd1144 }
-/* LiteralStringDelimiter */ .chroma .dl { color: #dd1144 }
-/* LiteralStringDoc */ .chroma .sd { color: #dd1144 }
-/* LiteralStringDouble */ .chroma .s2 { color: #dd1144 }
-/* LiteralStringEscape */ .chroma .se { color: #dd1144 }
-/* LiteralStringHeredoc */ .chroma .sh { color: #dd1144 }
-/* LiteralStringInterpol */ .chroma .si { color: #dd1144 }
-/* LiteralStringOther */ .chroma .sx { color: #dd1144 }
-/* LiteralStringRegex */ .chroma .sr { color: #009926 }
-/* LiteralStringSingle */ .chroma .s1 { color: #dd1144 }
-/* LiteralStringSymbol */ .chroma .ss { color: #990073 }
-/* LiteralNumber */ .chroma .m { color: #009999 }
-/* LiteralNumberBin */ .chroma .mb { color: #009999 }
-/* LiteralNumberFloat */ .chroma .mf { color: #009999 }
-/* LiteralNumberHex */ .chroma .mh { color: #009999 }
-/* LiteralNumberInteger */ .chroma .mi { color: #009999 }
-/* LiteralNumberIntegerLong */ .chroma .il { color: #009999 }
-/* LiteralNumberOct */ .chroma .mo { color: #009999 }
-/* Operator */ .chroma .o { color: #000000; font-weight: bold }
-/* OperatorWord */ .chroma .ow { color: #000000; font-weight: bold }
-/* Punctuation */ .chroma .p { }
-/* Comment */ .chroma .c { color: #999988; font-style: italic }
-/* CommentHashbang */ .chroma .ch { color: #999988; font-style: italic }
-/* CommentMultiline */ .chroma .cm { color: #999988; font-style: italic }
-/* CommentSingle */ .chroma .c1 { color: #999988; font-style: italic }
-/* CommentSpecial */ .chroma .cs { color: #999999; font-weight: bold; font-style: italic }
-/* CommentPreproc */ .chroma .cp { color: #999999; font-weight: bold; font-style: italic }
-/* CommentPreprocFile */ .chroma .cpf { color: #999999; font-weight: bold; font-style: italic }
-/* Generic */ .chroma .g { }
-/* GenericDeleted */ .chroma .gd { color: #000000; background-color: #ffdddd }
-/* GenericEmph */ .chroma .ge { color: #000000; font-style: italic }
-/* GenericError */ .chroma .gr { color: #aa0000 }
-/* GenericHeading */ .chroma .gh { color: #999999 }
-/* GenericInserted */ .chroma .gi { color: #000000; background-color: #ddffdd }
-/* GenericOutput */ .chroma .go { color: #888888 }
-/* GenericPrompt */ .chroma .gp { color: #555555 }
-/* GenericStrong */ .chroma .gs { font-weight: bold }
-/* GenericSubheading */ .chroma .gu { color: #aaaaaa }
-/* GenericTraceback */ .chroma .gt { color: #aa0000 }
-/* GenericUnderline */ .chroma .gl { text-decoration: underline }
-/* TextWhitespace */ .chroma .w { color: #bbbbbb }
diff --git a/docs/assets/search-data.js b/docs/assets/search-data.js
deleted file mode 100644
index 620fc380cf..0000000000
--- a/docs/assets/search-data.js
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-'use strict';
-
-(function () {
- const indexCfg = {{ with i18n "bookSearchConfig" }}
- {{ . }};
- {{ else }}
- {};
- {{ end }}
-
- indexCfg.doc = {
- id: 'id',
- field: ['title', 'content'],
- store: ['title', 'href', 'section'],
- };
-
- const index = FlexSearch.create('balance', indexCfg);
- window.bookSearchIndex = index;
-
- {{- $pages := where .Site.Pages "Kind" "in" (slice "page" "section") -}}
- {{- $pages = where $pages "Params.booksearchexclude" "!=" true -}}
- {{- $pages = where $pages "Content" "not in" (slice nil "") -}}
-
- {{ range $index, $page := $pages }}
- index.add({
- 'id': {{ $index }},
- 'href': '{{ $page.RelPermalink }}',
- 'title': {{ (partial "docs/simple-title" $page) | jsonify }},
- 'section': {{ (partial "docs/simple-title" $page.Parent) | jsonify }},
- 'content': {{ $page.Plain | jsonify }}
- });
- {{- end -}}
-})();
diff --git a/docs/babel.config.js b/docs/babel.config.js
new file mode 100644
index 0000000000..e00595dae7
--- /dev/null
+++ b/docs/babel.config.js
@@ -0,0 +1,3 @@
+module.exports = {
+ presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
+};
diff --git a/docs/config.toml b/docs/config.toml
deleted file mode 100644
index 5316a9ad9e..0000000000
--- a/docs/config.toml
+++ /dev/null
@@ -1,85 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-baseURL = '//paimon.apache.org/docs/master'
-languageCode = 'en-us'
-title = 'Apache Paimon'
-enableGitInfo = false
-theme = "book"
-pygmentsUseClasses = true
-
-[params]
- # Flag whether this is a stable version or not.
- # Used for the quickstart page.
- IsStable = false
-
- # Flag to indicate whether an outdated warning should be shown.
- ShowOutDatedWarning = false
-
- # This is the version referenced in the docs. Please only use these variables
- # to reference a specific Paimon version, because this is the only place where
- # we change the version for the complete docs when forking of a release branch
- # etc.
- # The full version string as referenced in Maven (e.g. 1.2.1)
- Version = "1.5-SNAPSHOT"
-
- # For stable releases, leave the bugfix version out (e.g. 1.2). For snapshot
- # release this should be the same as the regular version
- VersionTitle = "1.5-SNAPSHOT"
-
- # The branch for this version of Apache Paimon
- Branch = "master"
-
- # The most recent supported Apache Flink version
- FlinkVersion = "1.20"
-
- # The github repository for Apache Paimon
- Repo = "//github.com/apache/paimon"
-
- GithubRepo = "https://github.com/apache/paimon.git"
-
- TrinoGithubRepo = "https://github.com/apache/paimon-trino.git"
-
- PrestoGithubRepo = "https://github.com/apache/paimon-presto.git"
-
- # This suffix is appended to the Scala-dependent Maven artifact names
- ScalaVersion = "_2.12"
-
- ProjectHomepage = "//paimon.apache.org"
-
- # External links at the bottom
- # of the menu
- MenuLinks = [
- ["Project Homepage", "//paimon.apache.org"],
- ["JavaDocs", "//paimon.apache.org/docs/master/api/java/"],
- ]
-
- StableDocs = "https://paimon.apache.org/docs/1.4"
-
- PreviousDocs = [
- ["master", "https://paimon.apache.org/docs/master"],
- ["stable", "https://paimon.apache.org/docs/1.4"],
- ["1.4", "https://paimon.apache.org/docs/1.4"],
- ["1.3", "https://paimon.apache.org/docs/1.3"],
- ["1.2", "https://paimon.apache.org/docs/1.2"],
- ["1.1", "https://paimon.apache.org/docs/1.1"],
- ]
-
- BookSection = '/'
-
-[markup]
-[markup.goldmark.renderer]
- unsafe = true
diff --git a/docs/content/_index.md b/docs/content/_index.md
deleted file mode 100644
index a9f7113bc6..0000000000
--- a/docs/content/_index.md
+++ /dev/null
@@ -1,89 +0,0 @@
----
-title: Apache Paimon
-type: docs
-bookToc: false
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Apache Paimon
-
-Apache Paimon is a lake format for building Lakehouse Architecture for both streaming and batch
-operations. Paimon provides large-scale data lake storage for analytics, realtime streaming updates
-powered by LSM (Log-structured merge-tree) structure, and multimodal data management for AI workloads
-— all in a single unified format.
-
-## Large-Scale Data Lake
-
-Paimon is built for huge analytic datasets. A single table can contain tens of petabytes of data, and even
-these huge tables can be read efficiently without a distributed SQL engine.
-
-- **Time travel** enables reproducible queries that use exactly the same table snapshot, or lets users easily
- examine changes. Version rollback allows users to quickly correct problems by resetting tables to a good state.
-- **Scan planning is fast** — data files are pruned with partition and column-level stats, using table metadata.
- File Index (BloomFilter, Bitmap, Range Bitmap) and aggregate push-down further accelerate queries.
-- **Schema evolution** supports add, drop, update, or rename columns, and has no side-effects.
-- **Rich ecosystem** — adds tables to compute engines including Flink, Spark, Hive, Trino, Presto, StarRocks, and
- Doris, working just like a SQL table.
-- **Incremental Clustering** with z-order/hilbert/order sorting to optimize data layout at low cost.
-
-## Realtime Data Lake
-
-Paimon's Primary Key Table brings realtime streaming updates into the lake architecture, powered by the LSM
-(Log-structured merge-tree) structure.
-
-- **Large-scale streaming updates** with very high performance, typically through Flink Streaming.
-- **Multiple Merge Engines**: Deduplicate to keep last row, Partial Update to progressively complete records,
- Aggregation to aggregate values, or First Row to keep the earliest record — update records however you like.
-- **Multiple Table Modes**: Merge On Read (MOR), Copy On Write (COW), and Merge On Write (MOW) with Deletion Vectors
- for flexible read/write trade-offs.
-- **Changelog Producers** (None, Input, Lookup, Full Compaction) produce correct and complete changelog for merge
- engines, simplifying your streaming analytics.
-- **CDC Ingestion** from MySQL, Kafka, MongoDB, Pulsar, PostgreSQL, and Flink CDC with schema evolution support.
-
-## Multimodal Data Lake
-
-Paimon is a multimodal lakehouse for AI. Keep multimodal data, metadata, and embeddings in the same table and query
-them via vector search, full-text search, or SQL.
-
-- **Data Evolution** for efficient row-level updates and partial column changes without rewriting entire files — add
- new features (columns) as your application evolves, without copying existing data.
-- **Blob Table** for storing multimodal data (images, videos, audio, documents, model weights) with separated storage
- layout — blob data is stored in dedicated `.blob` files while metadata stays in standard columnar files.
-- **Global Index** with BTree Index for high-performance scalar lookups and Vector Index (DiskANN) for approximate
- nearest neighbor search.
-- **PyPaimon** native Python SDK with no JDK dependency, seamlessly integrating with the Python AI ecosystem
- including Ray, PyTorch, Pandas and PyArrow for data loading, training, and inference workflows.
-
-{{< columns >}}
-
-## Try Paimon
-
-If you're interested in playing around with Paimon, check out our
-quick start guide with [Flink]({{< ref "flink/quick-start" >}}) or [Spark]({{< ref "spark/quick-start" >}}). It provides a step by
-step introduction to the APIs and guides you through real applications.
-
-<--->
-
-## Get Help with Paimon
-
-If you get stuck, you can subscribe User Mailing List (user-subscribe@paimon.apache.org),
-Paimon tracks issues in GitHub and prefers to receive contributions as pull requests. You can also create an issue.
-
-{{< /columns >}}
diff --git a/docs/content/append-table/_index.md b/docs/content/append-table/_index.md
deleted file mode 100644
index 9848375b9d..0000000000
--- a/docs/content/append-table/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: "Table w/o PK"
-icon: <i class="fa fa-database title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 3
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/cdc-ingestion/_index.md b/docs/content/cdc-ingestion/_index.md
deleted file mode 100644
index b3d4dce2a0..0000000000
--- a/docs/content/cdc-ingestion/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: CDC Ingestion
-icon: <i class="fa fa-gear title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 91
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/concepts/_index.md b/docs/content/concepts/_index.md
deleted file mode 100644
index ceaed84997..0000000000
--- a/docs/content/concepts/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Concepts
-icon: <i class="fa fa-book title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 1
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/ecosystem/_index.md b/docs/content/ecosystem/_index.md
deleted file mode 100644
index 27620979d8..0000000000
--- a/docs/content/ecosystem/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Ecosystem
-icon: <i class="fa fa-gear title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 90
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/flink/_index.md b/docs/content/flink/_index.md
deleted file mode 100644
index 6ec757fa52..0000000000
--- a/docs/content/flink/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Engine Flink
-icon: <i class="fa fa-gear title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 5
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/iceberg/_index.md b/docs/content/iceberg/_index.md
deleted file mode 100644
index b9416216fe..0000000000
--- a/docs/content/iceberg/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Iceberg Metadata
-icon: <i class="fa fa-briefcase title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 98
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/learn-paimon/_index.md b/docs/content/learn-paimon/_index.md
deleted file mode 100644
index 1bac917107..0000000000
--- a/docs/content/learn-paimon/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Learn Paimon
-icon: <i class="fa fa-sitemap title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 100
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/maintenance/_index.md b/docs/content/maintenance/_index.md
deleted file mode 100644
index 197e93f0cc..0000000000
--- a/docs/content/maintenance/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Maintenance
-icon: <i class="fa fa-wrench title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 94
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/maintenance/configurations.md b/docs/content/maintenance/configurations.md
deleted file mode 100644
index ce127feb0d..0000000000
--- a/docs/content/maintenance/configurations.md
+++ /dev/null
@@ -1,91 +0,0 @@
----
-title: "Configurations"
-weight: 100
-type: docs
-aliases:
-- /maintenance/configurations.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Configuration
-
-### CoreOptions
-
-Core options for paimon.
-
-{{< generated/core_configuration >}}
-
-### CatalogOptions
-
-Options for paimon catalog.
-
-{{< generated/catalog_configuration >}}
-
-### HiveCatalogOptions
-
-Options for Hive catalog.
-
-{{< generated/hive_catalog_configuration >}}
-
-### HiveConnectorOptions
-
-Hive connector options for paimon.
-
-{{< generated/hive_connector_configuration >}}
-
-### JdbcCatalogOptions
-
-Options for Jdbc catalog.
-
-{{< generated/jdbc_catalog_configuration >}}
-
-### FlinkCatalogOptions
-
-Flink catalog options for paimon.
-
-{{< generated/flink_catalog_configuration >}}
-
-### FlinkConnectorOptions
-
-Flink connector options for paimon.
-
-{{< generated/flink_connector_configuration >}}
-
-### SparkCatalogOptions
-
-Spark catalog options for paimon.
-
-{{< generated/spark_catalog_configuration >}}
-
-### SparkConnectorOptions
-
-Spark connector options for paimon.
-
-{{< generated/spark_connector_configuration >}}
-
-### ORC Options
-
-{{< generated/orc_configuration >}}
-
-### RocksDB Options
-
-The following options allow users to finely adjust RocksDB for better performance. You can either specify them in table properties or in dynamic table hints.
-
-{{< generated/rocksdb_configuration >}}
diff --git a/docs/content/migration/_index.md b/docs/content/migration/_index.md
deleted file mode 100644
index 64ac60499a..0000000000
--- a/docs/content/migration/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Migration
-icon: <i class="fa fa-briefcase title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 97
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/program-api/_index.md b/docs/content/program-api/_index.md
deleted file mode 100644
index 0449fd8416..0000000000
--- a/docs/content/program-api/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Program API
-icon: <i class="fa fa-briefcase title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 95
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/project/_index.md b/docs/content/project/_index.md
deleted file mode 100644
index 8930c52b68..0000000000
--- a/docs/content/project/_index.md
+++ /dev/null
@@ -1,26 +0,0 @@
----
-title: Project
-icon: <i class="fa fa-sitemap title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-sectionBreak: true
-weight: 99
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/project/download.md b/docs/content/project/download.md
deleted file mode 100644
index 47105004fa..0000000000
--- a/docs/content/project/download.md
+++ /dev/null
@@ -1,150 +0,0 @@
----
-title: "Download"
-weight: 2
-type: docs
-aliases:
-- /project/download.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Download
-
-This documentation is a guide for downloading Paimon Jars.
-
-## Engine Jars
-
-{{< unstable >}}
-
-| Version | Jar |
-|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Flink 2.0 | Not yet released |
-| Flink 1.20 | [paimon-flink-1.20-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.20/{{< version >}}/) |
-| Flink 1.19 | [paimon-flink-1.19-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.19/{{< version >}}/) |
-| Flink 1.18 | [paimon-flink-1.18-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.18/{{< version >}}/) |
-| Flink 1.17 | [paimon-flink-1.17-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/{{< version >}}/) |
-| Flink 1.16 | [paimon-flink-1.16-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/{{< version >}}/) |
-| Flink Action | [paimon-flink-action-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/{{< version >}}/) |
-| Spark 4.1 | [paimon-spark-4.1_2.13-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.1_2.13/{{< version >}}/) |
-| Spark 4.0 | [paimon-spark-4.0_2.13-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0_2.13/{{< version >}}/) |
-| Spark 3.5 | [paimon-spark-3.5_2.12-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.12/{{< version >}}/) |
-| Spark 3.4 | [paimon-spark-3.4_2.12-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.4_2.12/{{< version >}}/) |
-| Spark 3.3 | [paimon-spark-3.3_2.12-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.3_2.12/{{< version >}}/) |
-| Spark 3.2 | [paimon-spark-3.2_2.12-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.2_2.12/{{< version >}}/) |
-| Hive 3.1 | [paimon-hive-connector-3.1-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/{{< version >}}/) |
-| Hive 2.3 | [paimon-hive-connector-2.3-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/{{< version >}}/) |
-| Hive 2.2 | [paimon-hive-connector-2.2-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/{{< version >}}/) |
-| Hive 2.1 | [paimon-hive-connector-2.1-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/{{< version >}}/) |
-| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/{{< version >}}/) | |
-| Trino 440 | [paimon-trino-440-{{< version >}}-plugin.tar.gz](https://repository.apache.org/content/repositories/snapshots/org/apache/paimon/paimon-trino-440/{{< version >}}/) |
-
-{{< /unstable >}}
-
-{{< stable >}}
-
-| Version | Jar |
-|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Flink 2.2 | [paimon-flink-2.2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.2/{{< version >}}/paimon-flink-2.2-{{< version >}}.jar) |
-| Flink 2.1 | [paimon-flink-2.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.1/{{< version >}}/paimon-flink-2.1-{{< version >}}.jar) |
-| Flink 2.0 | [paimon-flink-2.0-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.0/{{< version >}}/paimon-flink-2.0-{{< version >}}.jar) |
-| Flink 1.20 | [paimon-flink-1.20-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/{{< version >}}/paimon-flink-1.20-{{< version >}}.jar) |
-| Flink 1.19 | [paimon-flink-1.19-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.19/{{< version >}}/paimon-flink-1.19-{{< version >}}.jar) |
-| Flink 1.18 | [paimon-flink-1.18-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.18/{{< version >}}/paimon-flink-1.18-{{< version >}}.jar) |
-| Flink 1.17 | [paimon-flink-1.17-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/{{< version >}}/paimon-flink-1.17-{{< version >}}.jar) |
-| Flink 1.16 | [paimon-flink-1.16-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/{{< version >}}/paimon-flink-1.16-{{< version >}}.jar) |
-| Flink Action | [paimon-flink-action-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-action/{{< version >}}/paimon-flink-action-{{< version >}}.jar) |
-| Spark 4.1 | [paimon-spark-4.1_2.13-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.1_2.13/{{< version >}}/paimon-spark-4.1_2.13-{{< version >}}.jar) |
-| Spark 4.0 | [paimon-spark-4.0_2.13-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0_2.13/{{< version >}}/paimon-spark-4.0_2.13-{{< version >}}.jar) |
-| Spark 3.5 | [paimon-spark-3.5_2.12-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.12/{{< version >}}/paimon-spark-3.5_2.12-{{< version >}}.jar) |
-| Spark 3.4 | [paimon-spark-3.4_2.12-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.4_2.12/{{< version >}}/paimon-spark-3.4_2.12-{{< version >}}.jar) |
-| Spark 3.3 | [paimon-spark-3.3_2.12-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.3_2.12/{{< version >}}/paimon-spark-3.3_2.12-{{< version >}}.jar) |
-| Spark 3.2 | [paimon-spark-3.2_2.12-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.2_2.12/{{< version >}}/paimon-spark-3.2_2.12-{{< version >}}.jar) |
-| Hive 3.1 | [paimon-hive-connector-3.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/{{< version >}}/paimon-hive-connector-3.1-{{< version >}}.jar) |
-| Hive 2.3 | [paimon-hive-connector-2.3-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/{{< version >}}/paimon-hive-connector-2.3-{{< version >}}.jar) |
-| Hive 2.2 | [paimon-hive-connector-2.2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/{{< version >}}/paimon-hive-connector-2.2-{{< version >}}.jar) |
-| Hive 2.1 | [paimon-hive-connector-2.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/{{< version >}}/paimon-hive-connector-2.1-{{< version >}}.jar) |
-| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/{{< version >}}/paimon-hive-connector-2.1-cdh-6.3-{{< version >}}.jar) |
-| Trino | [Download from master](https://paimon.apache.org/docs/master/project/download/) |
-
-{{< /stable >}}
-
-## Filesystem Jars
-
-{{< unstable >}}
-
-| Version | Jar |
-|--------------|-----------------------------------------------------------------------------------------------------------------------------|
-| paimon-oss | [paimon-oss-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-oss/{{< version >}}/) |
-| paimon-jindo | [paimon-jindo-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-jindo/{{< version >}}/) |
-| paimon-s3 | [paimon-s3-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-s3/{{< version >}}/) |
-
-{{< /unstable >}}
-
-{{< stable >}}
-
-| Version | Jar |
-|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
-| paimon-oss | [paimon-oss-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/{{< version >}}/paimon-oss-{{< version >}}.jar) |
-| paimon-jindo | [paimon-jindo-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-jindo/{{< version >}}/paimon-jindo-{{< version >}}.jar) |
-| paimon-s3 | [paimon-s3-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/{{< version >}}/paimon-s3-{{< version >}}.jar) |
-
-{{< /stable >}}
-
-## API Jars
-
-{{< unstable >}}
-
-| Version | Jar |
-|---------------|-------------------------------------------------------------------------------------------------------------------------------|
-| paimon-bundle | [paimon-bundle-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-bundle/{{< version >}}/) |
-
-{{< /unstable >}}
-
-{{< stable >}}
-
-| Version | Jar |
-|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| paimon-bundle | [paimon-bundle-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-bundle/{{< version >}}/paimon-bundle-{{< version >}}.jar) |
-
-{{< /stable >}}
-
-{{< unstable >}}
-
-## MANIFEST.MF
-
-For unstable version, you can find git commit id in jar:
-
-```shell
-Manifest-Version: 1.0
-Implementation-Title: Paimon : Common
-Implementation-Version: {{< version >}}
-Specification-Vendor: The Apache Software Foundation
-Specification-Title: Paimon : Common
-Implementation-Vendor-Id: org.apache.paimon
-SCM-Branch: master
-Implementation-Vendor: The Apache Software Foundation
-SCM-Revision: c8b4772f3cb4b25b25537e1ab0775441c627bf1c
-Created-By: Apache Maven 3.2.5
-Build-Jdk: 1.8.0_301
-Specification-Version: {{< version >}}
-```
-
-The `SCM-Revision` git commit id.
-
-{{< /unstable >}}
\ No newline at end of file
diff --git a/docs/content/pypaimon/_index.md b/docs/content/pypaimon/_index.md
deleted file mode 100644
index b89939b6fc..0000000000
--- a/docs/content/pypaimon/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: PyPaimon
-icon: <i class="fa fa-briefcase title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 96
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/spark/_index.md b/docs/content/spark/_index.md
deleted file mode 100644
index 07128574b0..0000000000
--- a/docs/content/spark/_index.md
+++ /dev/null
@@ -1,25 +0,0 @@
----
-title: Engine Spark
-icon: <i class="fa fa-gear title maindish" aria-hidden="true"></i>
-bold: true
-bookCollapseSection: true
-weight: 6
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
diff --git a/docs/content/append-table/blob.md b/docs/docs/append-table/blob.mdx
similarity index 95%
rename from docs/content/append-table/blob.md
rename to docs/docs/append-table/blob.mdx
index b25205b168..4f0ce604ac 100644
--- a/docs/content/append-table/blob.md
+++ b/docs/docs/append-table/blob.mdx
@@ -1,10 +1,11 @@
---
title: "Blob Storage"
-weight: 7
-type: docs
-aliases:
-- /append-table/blob-storage.html
+sidebar_position: 7
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -67,7 +68,7 @@ This separation provides several benefits:
- Optimized file rolling based on blob size
- Better compression for regular columnar data
-For details about the blob file format structure, see [File Format - BLOB]({{< ref "concepts/spec/fileformat#blob" >}}).
+For details about the blob file format structure, see [File Format - BLOB](../concepts/spec/fileformat#blob).
## Storage Modes
@@ -89,35 +90,35 @@ This allows one table to mix raw-data BLOB fields, descriptor-only BLOB fields,
## Table Options
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 25%">Option</th>
- <th class="text-center" style="width: 8%">Required</th>
- <th class="text-center" style="width: 7%">Default</th>
- <th class="text-center" style="width: 10%">Type</th>
- <th class="text-center" style="width: 50%">Description</th>
+ <th className="text-left" style={{width: "25%"}}>Option</th>
+ <th className="text-center" style={{width: "8%"}}>Required</th>
+ <th className="text-center" style={{width: "7%"}}>Default</th>
+ <th className="text-center" style={{width: "10%"}}>Type</th>
+ <th className="text-center" style={{width: "50%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>blob-field</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>String</td>
<td>Specifies column names that should be stored as blob type. This is used when you want to treat a BYTES column as a BLOB. Fields listed in <code>blob-descriptor-field</code> or <code>blob-view-field</code> are also treated as BLOB fields.</td>
</tr>
<tr>
<td><h5>blob-as-descriptor</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">false</td>
+ <td style={{wordWrap: "break-word"}}>false</td>
<td>Boolean</td>
<td>Controls read output format for blob fields. When set to true, queries return serialized BlobDescriptor bytes; when false, queries return actual blob bytes. This option is dynamic and can be changed with <code>ALTER TABLE ... SET</code>.</td>
</tr>
<tr>
<td><h5>blob-descriptor-field</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>String</td>
<td>
Comma-separated field names treated as BLOB fields and stored as serialized <code>BlobDescriptor</code> bytes inline in normal data files.
@@ -129,7 +130,7 @@ This allows one table to mix raw-data BLOB fields, descriptor-only BLOB fields,
<tr>
<td><h5>blob-write-null-on-missing-file</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">false</td>
+ <td style={{wordWrap: "break-word"}}>false</td>
<td>Boolean</td>
<td>
When enabled for Flink writes, if a descriptor BLOB value references a file that does not exist, Paimon writes <code>NULL</code> for that value and logs a warning instead of failing when reading the descriptor.
@@ -138,7 +139,7 @@ This allows one table to mix raw-data BLOB fields, descriptor-only BLOB fields,
<tr>
<td><h5>blob-view-field</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>String</td>
<td>
Comma-separated field names treated as BLOB fields and stored as serialized <code>BlobViewStruct</code> bytes inline in normal data files.
@@ -149,7 +150,7 @@ This allows one table to mix raw-data BLOB fields, descriptor-only BLOB fields,
<tr>
<td><h5>blob-external-storage-field</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>String</td>
<td>
Comma-separated BLOB field names whose raw data should be written to external storage at write time.
@@ -160,7 +161,7 @@ This allows one table to mix raw-data BLOB fields, descriptor-only BLOB fields,
<tr>
<td><h5>blob-external-storage-path</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>String</td>
<td>
External storage path used by fields configured in <code>blob-external-storage-field</code>.
@@ -170,21 +171,21 @@ This allows one table to mix raw-data BLOB fields, descriptor-only BLOB fields,
<tr>
<td><h5>blob.target-file-size</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">(same as target-file-size)</td>
+ <td style={{wordWrap: "break-word"}}>(same as target-file-size)</td>
<td>MemorySize</td>
<td>Target size for blob files. When a blob file reaches this size, a new file is created. If not specified, uses the same value as <code>target-file-size</code>.</td>
</tr>
<tr>
<td><h5>row-tracking.enabled</h5></td>
<td>Yes*</td>
- <td style="word-wrap: break-word;">false</td>
+ <td style={{wordWrap: "break-word"}}>false</td>
<td>Boolean</td>
<td>Must be enabled for blob tables to support row-level operations.</td>
</tr>
<tr>
<td><h5>data-evolution.enabled</h5></td>
<td>Yes*</td>
- <td style="word-wrap: break-word;">false</td>
+ <td style={{wordWrap: "break-word"}}>false</td>
<td>Boolean</td>
<td>Must be enabled for blob tables to support schema evolution.</td>
</tr>
@@ -215,9 +216,10 @@ CREATE TABLE image_table (
### Creating a Table
-{{< tabs "blob-create-table" >}}
+<Tabs groupId="blob-create-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
-- Create a table with a blob field
-- Note: In Flink SQL, use BYTES type and specify blob-field option
@@ -231,9 +233,11 @@ CREATE TABLE image_table (
'blob-field' = 'image'
);
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
```sql
-- Create a table with a blob field
-- Note: In Spark SQL, use BINARY type and specify blob-field option
@@ -247,15 +251,17 @@ CREATE TABLE image_table (
'blob-field' = 'image'
);
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
### Inserting Blob Data
-{{< tabs "blob-insert" >}}
+<Tabs groupId="blob-insert">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
-- Insert data with inline blob bytes
INSERT INTO image_table VALUES (1, 'sample', X'89504E470D0A1A0A');
@@ -264,16 +270,19 @@ INSERT INTO image_table VALUES (1, 'sample', X'89504E470D0A1A0A');
INSERT INTO image_table
SELECT id, name, content FROM source_table;
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
```sql
-- Insert data with inline blob bytes
INSERT INTO image_table VALUES (1, 'sample', X'89504E470D0A1A0A');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
### Querying Blob Data
diff --git a/docs/content/append-table/bucketed.md b/docs/docs/append-table/bucketed.mdx
similarity index 90%
rename from docs/content/append-table/bucketed.md
rename to docs/docs/append-table/bucketed.mdx
index 04dc30699b..aac4f02c19 100644
--- a/docs/content/append-table/bucketed.md
+++ b/docs/docs/append-table/bucketed.mdx
@@ -1,10 +1,11 @@
---
title: "Bucketed"
-weight: 3
-type: docs
-aliases:
-- /append-table/bucketed.html
+sidebar_position: 3
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,8 +31,9 @@ You can define the `bucket` and `bucket-key` to get a bucketed append table.
Example to create bucketed append table:
-{{< tabs "create-bucketed-append" >}}
-{{< tab "Flink" >}}
+<Tabs groupId="create-bucketed-append">
+
+<TabItem value="flink" label="Flink">
```sql
CREATE TABLE my_table (
@@ -43,8 +45,10 @@ CREATE TABLE my_table (
'bucket-key' = 'product_id'
);
```
-{{< /tab >}}
-{{< /tabs >}}
+
+</TabItem>
+
+</Tabs>
## Data Skipping
@@ -94,7 +98,7 @@ Every record in the same bucket is ordered strictly, streaming read will transfe
the order of writing. To use this mode, you do not need to config special configurations, all the data will go into one
bucket as a queue.
-{{< img src="/img/for-queue.png">}}
+
**Streaming Read Order**
@@ -126,25 +130,25 @@ SELECT window_start, window_end, COUNT(`user`) FROM TABLE(
You can also enable [Flink Watermark alignment](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/event-time/generating_watermarks/#watermark-alignment-_beta_),
which will make sure no sources/splits/shards/partitions increase their watermarks too far ahead of the rest:
-<table class="configuration table table-bordered">
+<table className="configuration table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Key</th>
+ <th className="text-left" style={{width: "15%"}}>Default</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "55%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>scan.watermark.alignment.group</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>String</td>
<td>A group of sources to align watermarks.</td>
</tr>
<tr>
<td><h5>scan.watermark.alignment.max-drift</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>Duration</td>
<td>Maximal drift to align watermarks, before we pause consuming from the source/task/partition.</td>
</tr>
diff --git a/docs/content/append-table/data-evolution.md b/docs/docs/append-table/data-evolution.md
similarity index 98%
rename from docs/content/append-table/data-evolution.md
rename to docs/docs/append-table/data-evolution.md
index 2356ef365d..fe71e78415 100644
--- a/docs/content/append-table/data-evolution.md
+++ b/docs/docs/append-table/data-evolution.md
@@ -1,10 +1,8 @@
---
title: "Data Evolution"
-weight: 6
-type: docs
-aliases:
-- /append-table/data-evolution.html
+sidebar_position: 6
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -161,11 +159,11 @@ When reading: Paimon reads both the original data files and the new files contai
After writing, the files in `target_table` like below:
-{{< img src="/img/data-evolution.png">}}
+
When reading, the files with the same `first row id` will merge fields.
-{{< img src="/img/data-evolution2.png">}}
+
The advantage to the mode is:
diff --git a/docs/content/append-table/global-index.md b/docs/docs/append-table/global-index.mdx
similarity index 92%
rename from docs/content/append-table/global-index.md
rename to docs/docs/append-table/global-index.mdx
index 19354c7a40..4a8cc08778 100644
--- a/docs/content/append-table/global-index.md
+++ b/docs/docs/append-table/global-index.mdx
@@ -1,10 +1,11 @@
---
title: "Global Index"
-weight: 8
-type: docs
-aliases:
-- /append-table/global-index.html
+sidebar_position: 8
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -106,16 +107,18 @@ The legacy index type `lumina-vector-ann` is still accepted for existing tables
**Vector Search**
-{{< tabs "vector-search" >}}
+<Tabs groupId="vector-search">
+
+<TabItem value="spark-sql" label="Spark SQL">
-{{< tab "Spark SQL" >}}
```sql
-- Search for top-5 nearest neighbors
SELECT * FROM vector_search('my_table', 'embedding', array(1.0f, 2.0f, 3.0f), 5);
```
-{{< /tab >}}
-{{< tab "Flink SQL (Procedure)" >}}
+</TabItem>
+
+<TabItem value="flink-sql-procedure" label="Flink SQL (Procedure)">
Unlike Spark's table-valued function, Flink uses a `CALL` procedure to perform vector search.
The procedure returns JSON-serialized rows as strings.
@@ -139,9 +142,10 @@ CALL sys.vector_search(
);
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="java-api" label="Java API">
-{{< tab "Java API" >}}
```java
Table table = catalog.getTable(identifier);
@@ -162,9 +166,10 @@ try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)
});
}
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Full-Text Index
@@ -184,16 +189,19 @@ CALL sys.create_global_index(
**Full-Text Search**
-{{< tabs "fulltext-search" >}}
+<Tabs groupId="fulltext-search">
+
+<TabItem value="spark-sql" label="Spark SQL">
-{{< tab "Spark SQL" >}}
```sql
-- Search for top-10 documents matching the query
SELECT * FROM full_text_search('my_table', 'content', 'paimon lake format', 10);
```
-{{< /tab >}}
-{{< tab "Java API" >}}
+</TabItem>
+
+<TabItem value="java-api" label="Java API">
+
```java
Table table = catalog.getTable(identifier);
@@ -213,9 +221,11 @@ try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)
});
}
```
-{{< /tab >}}
-{{< tab "Python SDK" >}}
+</TabItem>
+
+<TabItem value="python-sdk" label="Python SDK">
+
```python
table = catalog.get_table('db.my_table')
@@ -234,6 +244,7 @@ table_read = read_builder.new_read()
pa_table = table_read.to_arrow(plan.splits())
print(pa_table)
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
diff --git a/docs/content/append-table/incremental-clustering.md b/docs/docs/append-table/incremental-clustering.mdx
similarity index 83%
rename from docs/content/append-table/incremental-clustering.md
rename to docs/docs/append-table/incremental-clustering.mdx
index 232ee13d7c..b8a2108d40 100644
--- a/docs/content/append-table/incremental-clustering.md
+++ b/docs/docs/append-table/incremental-clustering.mdx
@@ -1,10 +1,11 @@
---
title: "Incremental Clustering"
-weight: 2
-type: docs
-aliases:
-- /append-table/incremental-clustering.html
+sidebar_position: 2
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -26,7 +27,7 @@ under the License.
# Incremental Clustering
-Paimon currently supports ordering append tables using SFC (Space-Filling Curve)(see [sort compact]({{< ref "maintenance/dedicated-compaction#sort-compact" >}}) for more info).
+Paimon currently supports ordering append tables using SFC (Space-Filling Curve)(see [sort compact](../maintenance/dedicated-compaction#sort-compact) for more info).
The resulting data layout typically delivers better performance for queries that target clustering keys.
However, with the current SortCompaction, even when neither the data nor the clustering keys have changed,
each run still rewrites the entire dataset, which is extremely costly.
@@ -49,35 +50,35 @@ Incremental Clustering supports:
## Enable Incremental Clustering
To enable Incremental Clustering, the following configuration needs to be set for the table:
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Option</th>
- <th class="text-left" style="width: 10%">Value</th>
- <th class="text-left" style="width: 5%">Required</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Option</th>
+ <th className="text-left" style={{width: "10%"}}>Value</th>
+ <th className="text-left" style={{width: "5%"}}>Required</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "55%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>clustering.incremental</h5></td>
<td>true</td>
- <td style="word-wrap: break-word;">Yes</td>
+ <td style={{wordWrap: "break-word"}}>Yes</td>
<td>Boolean</td>
<td>Must be set to true to enable incremental clustering. Default is false.</td>
</tr>
<tr>
<td><h5>clustering.columns</h5></td>
<td>'clustering-columns'</td>
- <td style="word-wrap: break-word;">Yes</td>
+ <td style={{wordWrap: "break-word"}}>Yes</td>
<td>String</td>
<td>The clustering columns, in the format 'columnName1,columnName2'. It is not recommended to use partition keys as clustering keys.</td>
</tr>
<tr>
<td><h5>clustering.strategy</h5></td>
<td>'zorder' or 'hilbert' or 'order'</td>
- <td style="word-wrap: break-word;">No</td>
+ <td style={{wordWrap: "break-word"}}>No</td>
<td>String</td>
<td>The ordering algorithm used for clustering. If not set, It'll decided from the number of clustering columns. 'order' is used for 1 column, 'zorder' for less than 5 columns, and 'hilbert' for 5 or more columns.</td>
</tr>
@@ -93,20 +94,20 @@ Therefore, when Incremental Clustering is enabled, the table no longer supports
clustering and small-file merging must be performed exclusively via Incremental Clustering runs.
## Run Incremental Clustering
-{{< hint info >}}
+:::info
only support running Incremental Clustering in batch mode.
-{{< /hint >}}
+:::
To run a Incremental Clustering job, follow these instructions.
You don't need to specify any clustering-related parameters when running Incremental Clustering,
these options are already defined as table options. If you need to change clustering settings, please update the corresponding table options.
-{{< tabs "incremental-clustering" >}}
+<Tabs groupId="incremental-clustering">
-{{< tab "Spark SQL" >}}
+<TabItem value="spark-sql" label="Spark SQL">
Run the following sql:
@@ -120,15 +121,16 @@ CALL sys.compact(table => 'T')
-- run incremental clustering with full mode, this will recluster all data
CALL sys.compact(table => 'T', compact_strategy => 'full')
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
Run the following command to submit a incremental clustering job for the table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -142,7 +144,7 @@ Example: run incremental clustering
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse s3:///path/to/warehouse \
--database test_db \
@@ -160,9 +162,10 @@ Example: run incremental clustering
Note: write parallelism is set by `sink.parallelism`, if too big, may generate a large number of small files.
You can use `-D execution.runtime-mode=batch` or `-yD execution.runtime-mode=batch` (for the ON-YARN scenario) to use batch mode.
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Auto-Clustering For Historical Partition
While performing incremental clustering on recently active partitions, Paimon can automatically detect historical and
@@ -171,28 +174,28 @@ For those historical partitions that have not yet achieved optimal layout, Paimo
during the same operation, thereby improving their query performance.
To enable auto-clustering for historical partitions, the following configuration needs to be set for the table:
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Option</th>
- <th class="text-left" style="width: 10%">Value</th>
- <th class="text-left" style="width: 5%">Required</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Option</th>
+ <th className="text-left" style={{width: "10%"}}>Value</th>
+ <th className="text-left" style={{width: "5%"}}>Required</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "55%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>clustering.history-partition.idle-to-full-sort</h5></td>
<td>3d</td>
- <td style="word-wrap: break-word;">Yes</td>
+ <td style={{wordWrap: "break-word"}}>Yes</td>
<td>Duration</td>
<td>The duration after which a partition without new updates is considered a historical partition. Default is null.</td>
</tr>
<tr>
<td><h5>clustering.history-partition.limit</h5></td>
<td>5</td>
- <td style="word-wrap: break-word;">Yes</td>
+ <td style={{wordWrap: "break-word"}}>Yes</td>
<td>Integer</td>
<td>The limit of history partition number for automatically performing full clustering. Default value is 5.</td>
</tr>
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/append-table/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/append-table/index.md
index eacaa231ca..1b47be4a90 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/append-table/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Table w/o PK"
+sidebar_position: 3
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/append-table/overview.md b/docs/docs/append-table/overview.mdx
similarity index 89%
rename from docs/content/append-table/overview.md
rename to docs/docs/append-table/overview.mdx
index 8644106e83..59d3a785f4 100644
--- a/docs/content/append-table/overview.md
+++ b/docs/docs/append-table/overview.mdx
@@ -1,10 +1,11 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /append-table/overview.html
+sidebar_position: 1
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,8 +31,10 @@ If a table does not have a primary key defined, it is an append table. Compared
have the ability to directly receive changelogs. It cannot be directly updated with data through upsert. It can only
receive incoming data from append data.
-{{< tabs "create-append-table" >}}
-{{< tab "Flink" >}}
+<Tabs groupId="create-append-table">
+
+<TabItem value="flink" label="Flink">
+
```sql
CREATE TABLE my_table (
product_id BIGINT,
@@ -44,8 +47,10 @@ CREATE TABLE my_table (
-- 'file.compression.zstd-level' = '3'
);
```
-{{< /tab >}}
-{{< /tabs >}}
+
+</TabItem>
+
+</Tabs>
Batch write and batch read in typical application scenarios, similar to a regular Hive partition table, but compared to
the Hive table, it can bring:
@@ -111,7 +116,7 @@ In streaming write job, without bucket definition, there is no compaction in wri
`Compact Coordinator` to scan the small files and pass compaction task to `Compact Worker`. In streaming mode, if you
run insert sql in flink, the topology will be like this:
-{{< img src="/img/unaware-bucket-topo.png">}}
+
Do not worry about backpressure, compaction never backpressure.
@@ -130,7 +135,7 @@ for streaming reads:
stream read incremental only.
Similar to flink-kafka, order is not guaranteed by default, if your data has some sort of order requirement, you also
-need to consider defining a `bucket-key`, see [Bucketed Append]({{< ref "append-table/bucketed" >}})
+need to consider defining a `bucket-key`, see [Bucketed Append](./bucketed)
## Aggregate push down
@@ -161,8 +166,8 @@ perform file filtering. If the filtering effect is good, the query that would ha
milliseconds to complete the execution.
Often the data distribution is not always ideal for filtering, so can we sort the data by the field in `WHERE` condition?
-You can take a look at [Flink COMPACT Action]({{< ref "maintenance/dedicated-compaction#sort-compact" >}}),
-[Flink COMPACT Procedure]({{< ref "flink/procedures" >}}) or [Spark COMPACT Procedure]({{< ref "spark/procedures" >}}).
+You can take a look at [Flink COMPACT Action](../maintenance/dedicated-compaction#sort-compact),
+[Flink COMPACT Procedure](../flink/procedures) or [Spark COMPACT Procedure](../spark/procedures).
## Data Skipping By File Index
@@ -176,19 +181,19 @@ definition and can contain different types of indexes with multiple columns.
Different file indexes may be efficient in different scenarios. For example bloom filter may speed up query in point lookup
scenario. Using a bitmap may consume more space but can result in greater accuracy.
-* [BloomFilter]({{< ref "concepts/spec/fileindex#index-bloomfilter" >}}): `file-index.bloom-filter.columns`.
-* [Bitmap]({{< ref "concepts/spec/fileindex#index-bitmap" >}}): `file-index.bitmap.columns`.
-* [Range Bitmap]({{< ref "concepts/spec/fileindex#index-range-bitmap" >}}): `file-index.range-bitmap.columns`.
+* [BloomFilter](../concepts/spec/fileindex#index-bloomfilter): `file-index.bloom-filter.columns`.
+* [Bitmap](../concepts/spec/fileindex#index-bitmap): `file-index.bitmap.columns`.
+* [Range Bitmap](../concepts/spec/fileindex#index-range-bitmap): `file-index.range-bitmap.columns`.
If you want to add file index to existing table, without any rewrite, you can use `rewrite_file_index` procedure. Before
we use the procedure, you should config appropriate configurations in target table. You can use ALTER clause to config
`file-index.<filter-type>.columns` to the table.
-How to invoke: see [flink procedures]({{< ref "flink/procedures#procedures" >}})
+How to invoke: see [flink procedures](../flink/procedures)
## Row Level Operations
-Now, only Spark SQL supports DELETE & UPDATE & MERGE INTO, you can take a look at [Spark Write]({{< ref "spark/sql-write" >}}).
+Now, only Spark SQL supports DELETE & UPDATE & MERGE INTO, you can take a look at [Spark Write](../spark/sql-write).
Example:
```sql
diff --git a/docs/content/append-table/row-tracking.md b/docs/docs/append-table/row-tracking.md
similarity index 98%
rename from docs/content/append-table/row-tracking.md
rename to docs/docs/append-table/row-tracking.md
index c1c815c487..a6737becb1 100644
--- a/docs/content/append-table/row-tracking.md
+++ b/docs/docs/append-table/row-tracking.md
@@ -1,10 +1,8 @@
---
title: "Row Tracking"
-weight: 5
-type: docs
-aliases:
-- /append-table/row-tracking.html
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/append-table/vector.md b/docs/docs/append-table/vector.md
similarity index 99%
rename from docs/content/append-table/vector.md
rename to docs/docs/append-table/vector.md
index f028dd32c0..d50540144a 100644
--- a/docs/content/append-table/vector.md
+++ b/docs/docs/append-table/vector.md
@@ -1,10 +1,8 @@
---
title: "Vector Storage"
-weight: 7
-type: docs
-aliases:
-- /append-table/vector-storage.html
+sidebar_position: 7
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/cdc-ingestion/flink-cdc.md b/docs/docs/cdc-ingestion/flink-cdc.mdx
similarity index 89%
rename from docs/content/cdc-ingestion/flink-cdc.md
rename to docs/docs/cdc-ingestion/flink-cdc.mdx
index 775c069d2b..dee95692a8 100644
--- a/docs/content/cdc-ingestion/flink-cdc.md
+++ b/docs/docs/cdc-ingestion/flink-cdc.mdx
@@ -1,10 +1,11 @@
---
title: "Flink CDC"
-weight: 10
-type: docs
-aliases:
- - /cdc-ingestion/flink-cdc.html
+sidebar_position: 10
---
+
+import ConfigTable from '@site/src/components/ConfigTable';
+import cdcConfigurationHtml from '@site/generated/cdc_configuration.html';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -70,7 +71,7 @@ pipeline:
Pipeline Connector Options
----------------
-{{< generated/cdc_configuration >}}
+<ConfigTable html={cdcConfigurationHtml} />
Catalog Options
----------------
@@ -78,7 +79,7 @@ Catalog Options
Apart from the pipeline connector options described above, in the CDC yaml file you can also configure options that
starts with `catalog.properties.`. For example, `catalog.properties.warehouse` or `catalog.properties.metastore`. Such
options will have their prefix removed and the rest be regarded as catalog options. Please refer to the
-[Configurations]({{< ref "maintenance/configurations" >}}) section for catalog options available.
+[Configurations](../maintenance/configurations) section for catalog options available.
Usage Notes
--------
@@ -90,13 +91,13 @@ Usage Notes
Data Type Mapping
----------------
-<div class="wy-table-responsive">
-<table class="colwidths-auto docutils">
+<div className="wy-table-responsive">
+<table className="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left">Paimon type</th>
- <th class="text-left">CDC type</th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th className="text-left">Paimon type</th>
+ <th className="text-left">CDC type</th>
+ <th className="text-left" style={{width: "60%"}}>NOTE</th>
</tr>
</thead>
<tbody>
@@ -169,4 +170,3 @@ Data Type Mapping
</table>
</div>
-{{< top >}}
\ No newline at end of file
diff --git a/docs/content/concepts/spec/_index.md b/docs/docs/cdc-ingestion/index.md
similarity index 93%
rename from docs/content/concepts/spec/_index.md
rename to docs/docs/cdc-ingestion/index.md
index 3cfdc14040..37cc4e086e 100644
--- a/docs/content/concepts/spec/_index.md
+++ b/docs/docs/cdc-ingestion/index.md
@@ -1,8 +1,8 @@
---
-title: Specification
-bookCollapseSection: true
-weight: 11
+title: "CDC Ingestion"
+sidebar_position: 91
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/cdc-ingestion/kafka-cdc.md b/docs/docs/cdc-ingestion/kafka-cdc.mdx
similarity index 82%
rename from docs/content/cdc-ingestion/kafka-cdc.md
rename to docs/docs/cdc-ingestion/kafka-cdc.mdx
index 81ec7c18ff..ba59d36d82 100644
--- a/docs/content/cdc-ingestion/kafka-cdc.md
+++ b/docs/docs/cdc-ingestion/kafka-cdc.mdx
@@ -1,10 +1,12 @@
---
title: "Kafka CDC"
-weight: 3
-type: docs
-aliases:
-- /cdc-ingestion/kafka-cdc.html
+sidebar_position: 3
---
+
+import ConfigTable from '@site/src/components/ConfigTable';
+import kafkaSyncTableHtml from '@site/generated/kafka_sync_table.html';
+import kafkaSyncDatabaseHtml from '@site/generated/kafka_sync_database.html';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -35,11 +37,11 @@ flink-sql-connector-kafka-*.jar
## Supported Formats
Flink provides several Kafka CDC formats: Canal Json, Debezium Json, Debezium Avro, Ogg Json, Maxwell Json and Normal Json.
If a message in a Kafka topic is a change event captured from another database using the Change Data Capture (CDC) tool, then you can use the Paimon Kafka CDC. Write the INSERT, UPDATE, DELETE messages parsed into the paimon table.
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left">Formats</th>
- <th class="text-left">Supported</th>
+ <th className="text-left">Formats</th>
+ <th className="text-left">Supported</th>
</tr>
</thead>
<tbody>
@@ -74,7 +76,8 @@ If a message in a Kafka topic is a change event captured from another database u
</tbody>
</table>
-{{< hint info >}}
+:::info
+
The JSON sources possibly missing some information. For example, Ogg and Maxwell format standards don't contain field
types; When you write JSON sources into Flink Kafka sink, it will only reserve data and row type and drop other information.
The synchronization job will try best to handle the problem as follows:
@@ -84,17 +87,18 @@ json has this field, or Paimon will use 'STRING' type.
3. If missing database name or table name, you cannot do database synchronization, but you can still do table synchronization.
4. If missing primary keys, the job might create non primary key table. You can set primary keys when submit job in table
synchronization.
-{{< /hint >}}
+
+:::
## Synchronizing Tables
-By using [KafkaSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Kafka's one topic into one Paimon table.
+By using [KafkaSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Kafka's one topic into one Paimon table.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_table \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -110,7 +114,7 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/kafka_sync_table >}}
+<ConfigTable html={kafkaSyncTableHtml} />
If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified Kafka topic's tables,it gets the earliest non-DDL data parsing schema from topic. If the Paimon table already exists, its schema will be compared against the schema of all specified Kafka topic's tables.
@@ -118,7 +122,7 @@ Example 1:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -162,7 +166,7 @@ Then you can submit synchronization job:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -176,7 +180,7 @@ For some append data (such as log data), it can be treated as special CDC data w
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -194,13 +198,13 @@ For some append data (such as log data), it can be treated as special CDC data w
## Synchronizing Databases
-By using [KafkaSyncDatabaseAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the multi topic or one topic into one Paimon database.
+By using [KafkaSyncDatabaseAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the multi topic or one topic into one Paimon database.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_database \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -224,7 +228,7 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/kafka_sync_database >}}
+<ConfigTable html={kafkaSyncDatabaseHtml} />
This action will build a single combined sink for all tables. For each Kafka topic's table to be synchronized, if the
corresponding Paimon table does not exist, this action will automatically create the table, and its schema will be derived
@@ -237,7 +241,7 @@ Synchronization from one Kafka topic to Paimon database.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -257,7 +261,7 @@ Synchronization from multiple Kafka topics to Paimon database.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -276,13 +280,13 @@ Synchronization from multiple Kafka topics to Paimon database.
There are some useful options to build Flink Kafka Source, but they are not provided by flink-kafka-connector document. They are:
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left">Key</th>
- <th class="text-left">Default</th>
- <th class="text-left">Type</th>
- <th class="text-left">Description</th>
+ <th className="text-left">Key</th>
+ <th className="text-left">Default</th>
+ <th className="text-left">Type</th>
+ <th className="text-left">Description</th>
</tr>
</thead>
<tbody>
@@ -297,7 +301,7 @@ There are some useful options to build Flink Kafka Source, but they are not prov
## Debezium-bson
-The debezium-bson format is one of the formats supported by <a href="{{< ref "/cdc-ingestion/kafka-cdc" >}}">Kafka CDC</a>.
+The debezium-bson format is one of the formats supported by [Kafka CDC](./kafka-cdc).
It is the format obtained by collecting mongodb through debezium, which is similar to
<a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/table/formats/debezium/">debezium-json</a> format.
However, MongoDB does not have a fixed schema, and the field types of each document may be different, so the before/after fields
@@ -309,11 +313,13 @@ MongoDB BSON Jar can be downloaded from the [Maven repository](https://mvnreposi
bson-*.jar
```
-{{< hint info >}}
+:::info
+
The debezium bson format requires insert/update/delete event messages include the full document, and include a field that represents the state of the document before the change.
This requires setting debezium's capture.mode to change_streams_update_full_with_pre_image and [capture.mode.full.update.type](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-capture-mode-full-update-type) to post_image.
Before version 6.0 of MongoDB, it was not possible to obtain 'Update Before' information. Therefore, using the id field in the Kafka Key as 'Update before' information
-{{< /hint >}}
+
+:::
Here is a simple example for an update operation captured from a Mongodb customers collection in JSON format:
@@ -384,12 +390,12 @@ Because the schema field of the event message does not have the field informatio
Below is a list of top-level field BsonValue conversion examples:
-<table class="configuration table table-bordered">
+<table className="configuration table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">BsonValue Type</th>
- <th class="text-left" style="width: 40%">Json Value</th>
- <th class="text-left" style="width: 40%">Conversion Result String</th>
+ <th className="text-left" style={{width: "20%"}}>BsonValue Type</th>
+ <th className="text-left" style={{width: "40%"}}>Json Value</th>
+ <th className="text-left" style={{width: "40%"}}>Conversion Result String</th>
</tr>
</thead>
<tbody>
@@ -408,7 +414,7 @@ Below is a list of top-level field BsonValue conversion examples:
<td>
<ul>
<li>1735934393769</li>
- <li>{"$numberLong": "1735934393769"}</li>
+ <li>{'\{'}"$numberLong": "1735934393769"{'\}'}</li>
</ul>
</td>
<td>"1735934393769"</td>
@@ -417,10 +423,10 @@ Below is a list of top-level field BsonValue conversion examples:
<td><h5>BsonDouble</h5></td>
<td>
<ul>
- <li>{"$numberDouble": "3.14"}</li>
- <li>{"$numberDouble": "NaN"}</li>
- <li>{"$numberDouble": "Infinity"}</li>
- <li>{"$numberDouble": "-Infinity"}</li>
+ <li>{'\{'}"$numberDouble": "3.14"{'\}'}</li>
+ <li>{'\{'}"$numberDouble": "NaN"{'\}'}</li>
+ <li>{'\{'}"$numberDouble": "Infinity"{'\}'}</li>
+ <li>{'\{'}"$numberDouble": "-Infinity"{'\}'}</li>
</ul>
</td>
<td>
@@ -449,17 +455,17 @@ Below is a list of top-level field BsonValue conversion examples:
</tr>
<tr>
<td><h5>BsonArray</h5></td>
- <td>[1,2,{"$numberLong": "1735934393769"}]</td>
+ <td>[1,2,{'\{'}"$numberLong": "1735934393769"{'\}'}]</td>
<td>"[1,2,1735934393769]"</td>
</tr>
<tr>
<td><h5>BsonObjectId</h5></td>
- <td>{"$oid": "596e275826f08b2730779e1f"}</td>
+ <td>{'\{'}"$oid": "596e275826f08b2730779e1f"{'\}'}</td>
<td>"596e275826f08b2730779e1f"</td>
</tr>
<tr>
<td><h5>BsonDateTime</h5></td>
- <td>{"$date": 1735934393769 }</td>
+ <td>{'\{'}"$date": 1735934393769 {'\}'}</td>
<td>"1735934393769"</td>
</tr>
<tr>
@@ -469,25 +475,25 @@ Below is a list of top-level field BsonValue conversion examples:
</tr>
<tr>
<td><h5>BsonUndefined</h5></td>
- <td>{"$undefined": true}</td>
+ <td>{'\{'}"$undefined": true{'\}'}</td>
<td>null</td>
</tr>
<tr>
<td><h5>BsonBinary</h5></td>
- <td>{"$binary": "uE2/4v5MSVOiJZkOo3APKQ==", "$type": "0"}</td>
+ <td>{'\{'}"$binary": "uE2/4v5MSVOiJZkOo3APKQ==", "$type": "0"{'\}'}</td>
<td>"uE2/4v5MSVOiJZkOo3APKQ=="</td>
</tr>
<tr>
<td><h5>BsonBinary(type=UUID)</h5></td>
- <td>{"$binary": "uE2/4v5MSVOiJZkOo3APKQ==", "$type": "4"}</td>
+ <td>{'\{'}"$binary": "uE2/4v5MSVOiJZkOo3APKQ==", "$type": "4"{'\}'}</td>
<td>"b84dbfe2-fe4c-4953-a225-990ea3700f29"</td>
</tr>
<tr>
<td><h5>BsonDecimal128</h5></td>
<td>
<ul>
- <li>{"$numberDecimal": "3.14"}</li>
- <li>{"$numberDecimal": "NaN"}</li>
+ <li>{'\{'}"$numberDecimal": "3.14"{'\}'}</li>
+ <li>{'\{'}"$numberDecimal": "NaN"{'\}'}</li>
</ul>
</td>
<td>
@@ -499,60 +505,60 @@ Below is a list of top-level field BsonValue conversion examples:
</tr>
<tr>
<td><h5>BsonRegularExpression</h5></td>
- <td>{"$regularExpression": {"pattern": "^pass$", "options": "i"}}</td>
+ <td>{'\{'}"$regularExpression": {'\{'}"pattern": "^pass$", "options": "i"{'\}'}{'\}'}</td>
<td>"/^pass$/i"</td>
</tr>
<tr>
<td><h5>BsonSymbol</h5></td>
- <td>{"$symbol": "symbol"}</td>
+ <td>{'\{'}"$symbol": "symbol"{'\}'}</td>
<td>"symbol"</td>
</tr>
<tr>
<td><h5>BsonTimestamp</h5></td>
- <td>{"$timestamp": {"t": 1736997330, "i": 2}}</td>
+ <td>{'\{'}"$timestamp": {'\{'}"t": 1736997330, "i": 2{'\}'}{'\}'}</td>
<td>"1736997330"</td>
</tr>
<tr>
<td><h5>BsonMinKey</h5></td>
- <td>{"$minKey": 1}</td>
+ <td>{'\{'}"$minKey": 1{'\}'}</td>
<td>"BsonMinKey"</td>
</tr>
<tr>
<td><h5>BsonMaxKey</h5></td>
- <td>{"$maxKey": 1}</td>
+ <td>{'\{'}"$maxKey": 1{'\}'}</td>
<td>"BsonMaxKey"</td>
</tr>
<tr>
<td><h5>BsonJavaScript</h5></td>
- <td>{"$code": "function(){}"}</td>
- <td>"function(){}"</td>
+ <td>{'\{'}"$code": "function(){'\{'}}"{'\}'}</td>
+ <td>"function(){'\{'}}"</td>
</tr>
<tr>
<td><h5>BsonJavaScriptWithScope</h5></td>
- <td>{"$code": "function(){}", "$scope": {"name": "Anne"}}</td>
- <td>'{"$code": "function(){}", "$scope": {"name": "Anne"}}'</td>
+ <td>{'\{'}"$code": "function(){'\{'}}", "$scope": {'\{'}"name": "Anne"{'\}'}{'\}'}</td>
+ <td>'{'\{'}"$code": "function(){'\{'}}", "$scope": {'\{'}"name": "Anne"{'\}'}{'\}'}'</td>
</tr>
<tr>
<td><h5>BsonDocument</h5></td>
<td>
<pre>
-{
- "decimalPi": {"$numberDecimal": "3.14"},
- "doublePi": {"$numberDouble": "3.14"},
- "doubleNaN": {"$numberDouble": "NaN"},
- "decimalNaN": {"$numberDecimal": "NaN"},
- "long": {"$numberLong": "100"},
+{'\{'}
+ "decimalPi": {'\{'}"$numberDecimal": "3.14"{'\}'},
+ "doublePi": {'\{'}"$numberDouble": "3.14"{'\}'},
+ "doubleNaN": {'\{'}"$numberDouble": "NaN"{'\}'},
+ "decimalNaN": {'\{'}"$numberDecimal": "NaN"{'\}'},
+ "long": {'\{'}"$numberLong": "100"{'\}'},
"bool": true,
"array": [
- {"$numberInt": "1"},
- {"$numberLong": "2"}
+ {'\{'}"$numberInt": "1"{'\}'},
+ {'\{'}"$numberLong": "2"{'\}'}
]
-}
+{'\}'}
</pre>
</td>
<td>
<pre>
-'{
+'{'\{'}
"decimalPi":3.14,
"doublePi":3.14,
"doubleNaN":"NaN",
@@ -560,7 +566,7 @@ Below is a list of top-level field BsonValue conversion examples:
"long":100,
"bool":true,
"array":[1,2]
-}'
+{'\}'}'
</pre>
</td>
</tr>
diff --git a/docs/content/cdc-ingestion/mongo-cdc.md b/docs/docs/cdc-ingestion/mongo-cdc.md
similarity index 85%
rename from docs/content/cdc-ingestion/mongo-cdc.md
rename to docs/docs/cdc-ingestion/mongo-cdc.md
index 2cbd934343..4411f03c84 100644
--- a/docs/content/cdc-ingestion/mongo-cdc.md
+++ b/docs/docs/cdc-ingestion/mongo-cdc.md
@@ -1,10 +1,19 @@
---
title: "Mongo CDC"
-weight: 4
-type: docs
-aliases:
-- /cdc-ingestion/mongo-cdc.html
+sidebar_position: 4
---
+
+```mdx-code-block
+import ConfigTable from '@site/src/components/ConfigTable';
+import mongodbSyncTableHtml from '@site/generated/mongodb_sync_table.html';
+import mongodbOperatorHtml from '@site/generated/mongodb_operator.html';
+import mongodbFunctionsHtml from '@site/generated/mongodb_functions.html';
+import mongodbPathExampleHtml from '@site/generated/mongodb_path_example.html';
+import mongodbSyncDatabaseHtml from '@site/generated/mongodb_sync_database.html';
+```
+
+
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -33,19 +42,21 @@ under the License.
|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.5.0 | <li> <a href="https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-mongodb-cdc/3.5.0/flink-sql-connector-mongodb-cdc-3.5.0.jar">flink-sql-connector-mongodb-cdc-3.5.0.jar</a> |
-{{< hint danger >}}
+:::danger
+
Only CDC 3.5.0 or above is supported.
-{{< /hint >}}
+
+:::
## Synchronizing Tables
-By using [MongoDBSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mongodb/MongoDBSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one collection from MongoDB into one Paimon table.
+By using [MongoDBSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/mongodb/MongoDBSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one collection from MongoDB into one Paimon table.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mongodb_sync_table \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -57,7 +68,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/mongodb_sync_table >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mongodbSyncTableHtml} />
+```
+```
Here are a few points to take note of:
@@ -73,12 +88,20 @@ Here are a few points to take note of:
On the contrary, when `default.id.generation` is set to false, the MongoDB CDC source retains the original `_id` structure, without any additional processing. This mode offers users the flexibility to work with the raw `_id` format as provided by MongoDB, preserving any nested elements like `$oid`.
The choice between the two hinges on the user's preference: the former for a cleaner, simplified `_id` and the latter for a direct representation of MongoDB's `_id` structure.
-{{< generated/mongodb_operator >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mongodbOperatorHtml} />
+```
+```
Functions can be invoked at the tail end of a path - the input to a function is the output of the path expression. The function output is dictated by the function itself.
-{{< generated/mongodb_functions >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mongodbFunctionsHtml} />
+```
+```
Path Examples
```json
@@ -121,7 +144,11 @@ Path Examples
}
```
-{{< generated/mongodb_path_example >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mongodbPathExampleHtml} />
+```
+```
1. The synchronized table is required to have its primary key set as `_id`.
This is because MongoDB's change events are recorded before updates in messages.
@@ -138,7 +165,7 @@ Example 1: synchronize collection into one Paimon table
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mongodb_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -161,7 +188,7 @@ Example 2: Synchronize collection into a Paimon table according to the specified
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mongodb_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -184,13 +211,13 @@ Example 2: Synchronize collection into a Paimon table according to the specified
## Synchronizing Databases
-By using [MongoDBSyncDatabaseAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mongodb/MongoDBSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the whole MongoDB database into one Paimon database.
+By using [MongoDBSyncDatabaseAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/mongodb/MongoDBSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the whole MongoDB database into one Paimon database.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mongodb_sync_database \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -205,7 +232,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/mongodb_sync_database >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mongodbSyncDatabaseHtml} />
+```
+```
All collections to be synchronized need to set _id as the primary key.
For each MongoDB collection to be synchronized, if the corresponding Paimon table does not exist, this action will automatically create the table.
@@ -216,7 +247,7 @@ Example 1: synchronize entire database
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mongodb_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -236,7 +267,7 @@ Example 2: Synchronize the specified table.
```bash
<FLINK_HOME>/bin/flink run \
--fromSavepoint savepointPath \
-/path/to/paimon-flink-action-{{< version >}}.jar \
+/path/to/paimon-flink-action-@@VERSION@@.jar \
mongodb_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
diff --git a/docs/content/cdc-ingestion/mysql-cdc.md b/docs/docs/cdc-ingestion/mysql-cdc.md
similarity index 86%
rename from docs/content/cdc-ingestion/mysql-cdc.md
rename to docs/docs/cdc-ingestion/mysql-cdc.md
index f35db99e1c..6ab31992b9 100644
--- a/docs/content/cdc-ingestion/mysql-cdc.md
+++ b/docs/docs/cdc-ingestion/mysql-cdc.md
@@ -1,10 +1,16 @@
---
title: "Mysql CDC"
-weight: 2
-type: docs
-aliases:
-- /cdc-ingestion/mysql-cdc.html
+sidebar_position: 2
---
+
+```mdx-code-block
+import ConfigTable from '@site/src/components/ConfigTable';
+import mysqlSyncTableHtml from '@site/generated/mysql_sync_table.html';
+import mysqlSyncDatabaseHtml from '@site/generated/mysql_sync_database.html';
+```
+
+
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,25 +36,27 @@ Paimon supports synchronizing changes from different databases using change data
## Prepare CDC Bundled Jar
-Download `CDC Bundled Jar` and put them under <FLINK_HOME>/lib/.
+Download `CDC Bundled Jar` and put them under \<FLINK_HOME>/lib/.
| Version | Bundled Jar |
|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.5.0 | <li> <a href="https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-mysql-cdc/3.5.0/flink-sql-connector-mysql-cdc-3.5.0.jar">flink-sql-connector-mysql-cdc-3.5.0.jar</a> <li> <a href="https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar">mysql-connector-java-8.0.27.jar</a> |
-{{< hint danger >}}
+:::danger
+
Only CDC 3.5.0 or above is supported.
-{{< /hint >}}
+
+:::
## Synchronizing Tables
-By using [MySqlSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from MySQL into one Paimon table.
+By using [MySqlSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from MySQL into one Paimon table.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_table \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -63,7 +71,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/mysql_sync_table >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mysqlSyncTableHtml} />
+```
+```
If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified MySQL tables. If the Paimon table already exists, its schema will be compared against the schema of all specified MySQL tables.
@@ -71,7 +83,7 @@ If the Paimon table you specify does not exist, this action will automatically c
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -102,7 +114,7 @@ table 'source_table' is split into database 'source_db1', 'source_db2' ..., then
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -124,13 +136,13 @@ table 'source_table' is split into database 'source_db1', 'source_db2' ..., then
## Synchronizing Databases
-By using [MySqlSyncDatabaseAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the whole MySQL database into one Paimon database.
+By using [MySqlSyncDatabaseAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the whole MySQL database into one Paimon database.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_database \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -150,7 +162,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/mysql_sync_database >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={mysqlSyncDatabaseHtml} />
+```
+```
Only tables with primary keys will be synchronized.
@@ -160,7 +176,7 @@ For each MySQL table to be synchronized, if the corresponding Paimon table does
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -182,7 +198,7 @@ under database `source_db`. The command to submit the job looks like:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -209,7 +225,7 @@ The command to recover from previous snapshot and add new tables to synchronize
```bash
<FLINK_HOME>/bin/flink run \
--fromSavepoint savepointPath \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -223,9 +239,11 @@ The command to recover from previous snapshot and add new tables to synchronize
--including_tables 'product|user|address|order|custom'
```
-{{< hint info >}}
+:::info
+
You can set `--mode combined` to enable synchronizing newly added tables without restarting job.
-{{< /hint >}}
+
+:::
### Example 3: synchronize and merge multiple shards
@@ -234,7 +252,7 @@ synchronize all the `db.+.tbl.+` into tables `test_db.tbl1`, `test_db.tbl2` ...
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
mysql_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -253,16 +271,18 @@ synchronize all the `db.+.tbl.+` into tables `test_db.tbl1`, `test_db.tbl2` ...
By setting database-name to a regular expression, the synchronization job will capture all tables under matched databases
and merge tables of the same name into one table.
-{{< hint info >}}
+:::info
+
You can set `--merge_shards false` to prevent merging shards. The synchronized tables will be named to 'databaseName_tableName'
to avoid potential name conflict.
-{{< /hint >}}
+
+:::
## FAQ
1. Chinese characters in records ingested from MySQL are garbled.
-* Try to set `env.java.opts: -Dfile.encoding=UTF-8` in `flink-conf.yaml`(Flink version < 1.19) or `config.yaml`(Flink version >= 1.19)
+* Try to set `env.java.opts: -Dfile.encoding=UTF-8` in `flink-conf.yaml`(Flink version \< 1.19) or `config.yaml`(Flink version >= 1.19)
(the option is changed to `env.java.opts.all` since Flink-1.17).
2. Synchronize MySQL table and column comment.
diff --git a/docs/content/cdc-ingestion/overview.md b/docs/docs/cdc-ingestion/overview.mdx
similarity index 91%
rename from docs/content/cdc-ingestion/overview.md
rename to docs/docs/cdc-ingestion/overview.mdx
index 61a1ff8975..5ff105b656 100644
--- a/docs/content/cdc-ingestion/overview.md
+++ b/docs/docs/cdc-ingestion/overview.mdx
@@ -1,10 +1,12 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /cdc-ingestion/overview.html
+sidebar_position: 1
---
+
+import ConfigTable from '@site/src/components/ConfigTable';
+import temporalFunctionsHtml from '@site/generated/temporal_functions.html';
+import otherFunctionsHtml from '@site/generated/other_functions.html';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -33,7 +35,7 @@ We currently support the following sync ways:
1. MySQL Synchronizing Table: synchronize one or multiple tables from MySQL into one Paimon table.
2. MySQL Synchronizing Database: synchronize the whole MySQL database into one Paimon database.
-3. [Program API Sync]({{< ref "/program-api/flink-api#cdc-ingestion-table" >}}): synchronize your custom DataStream input into one Paimon table.
+3. [Program API Sync](../program-api/flink-api#cdc-ingestion-table): synchronize your custom DataStream input into one Paimon table.
4. Kafka Synchronizing Table: synchronize one Kafka topic's table into one Paimon table.
5. Kafka Synchronizing Database: synchronize one Kafka topic containing multiple tables or multiple topics containing one table each into one Paimon database.
6. MongoDB Synchronizing Collection: synchronize one Collection from MongoDB into one Paimon table.
@@ -44,21 +46,21 @@ We currently support the following sync ways:
## What is Schema Evolution
Suppose we have a MySQL table named `tableA`, it has three fields: `field_1`, `field_2`, `field_3`. When we want to load
-this MySQL table to Paimon, we can do this in Flink SQL, or use [MySqlSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction).
+this MySQL table to Paimon, we can do this in Flink SQL, or use [MySqlSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction).
**Flink SQL:**
In Flink SQL, if we change the table schema of the MySQL table after the ingestion, the table schema change will not be synchronized to Paimon.
-{{< img src="/img/cdc-ingestion-flinksql.png">}}
+
**MySqlSyncTableAction:**
-In [MySqlSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction),
+In [MySqlSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction),
if we change the table schema of the MySQL table after the ingestion, the table schema change will be synchronized to Paimon,
and the data of `field_4` which is newly added will be synchronized to Paimon too.
-{{< img src="/img/cdc-ingestion-schema-evolution.png">}}
+
## Schema Change Evolution
@@ -85,7 +87,7 @@ behaviors of `RENAME TABLE` and `DROP COLUMN` will be ignored, `RENAME COLUMN` w
Temporal functions can convert date and epoch time to another form. A common use case is to generate partition values.
-{{< generated/temporal_functions >}}
+<ConfigTable html={temporalFunctionsHtml} />
The data type of the temporal-column can be one of the following cases:
1. DATE, DATETIME or TIMESTAMP.
@@ -110,7 +112,7 @@ first day of a week is Sunday).
### Other Functions
-{{< generated/other_functions >}}
+<ConfigTable html={otherFunctionsHtml} />
## Special Data Type Mapping
diff --git a/docs/content/cdc-ingestion/postgres-cdc.md b/docs/docs/cdc-ingestion/postgres-cdc.md
similarity index 86%
rename from docs/content/cdc-ingestion/postgres-cdc.md
rename to docs/docs/cdc-ingestion/postgres-cdc.md
index 3d6f8f356e..d8eef18410 100644
--- a/docs/content/cdc-ingestion/postgres-cdc.md
+++ b/docs/docs/cdc-ingestion/postgres-cdc.md
@@ -1,10 +1,15 @@
---
title: "Postgres CDC"
-weight: 2
-type: docs
-aliases:
-- /cdc-ingestion/postgres-cdc.html
+sidebar_position: 2
---
+
+```mdx-code-block
+import ConfigTable from '@site/src/components/ConfigTable';
+import postgresSyncTableHtml from '@site/generated/postgres_sync_table.html';
+```
+
+
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,25 +35,27 @@ Paimon supports synchronizing changes from different databases using change data
## Prepare CDC Bundled Jar
-Download `CDC Bundled Jar` and put them under <FLINK_HOME>/lib/.
+Download `CDC Bundled Jar` and put them under \<FLINK_HOME>/lib/.
| Version | Bundled Jar |
|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.5.0 | <li> <a href="https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-postgres-cdc/3.5.0/flink-sql-connector-postgres-cdc-3.5.0.jar">flink-sql-connector-postgres-cdc-3.5.0.jar</a> |
-{{< hint danger >}}
+:::danger
+
Only CDC 3.5.0 or above is supported.
-{{< /hint >}}
+
+:::
## Synchronizing Tables
-By using [PostgresSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/postgres/PostgresSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from PostgreSQL into one Paimon table.
+By using [PostgresSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/postgres/PostgresSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from PostgreSQL into one Paimon table.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
postgres_sync_table \
--warehouse <warehouse_path> \
--database <database_name> \
@@ -63,7 +70,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon_table_sink_conf> [--table_conf <paimon_table_sink_conf> ...]]
```
-{{< generated/postgres_sync_table >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={postgresSyncTableHtml} />
+```
+```
If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified PostgreSQL tables. If the Paimon table already exists, its schema will be compared against the schema of all specified PostgreSQL tables.
@@ -71,7 +82,7 @@ Example 1: synchronize tables into one Paimon table
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
postgres_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -104,7 +115,7 @@ table 'source_table' is split into schema 'source_schema1', 'source_schema2' ...
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
postgres_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
diff --git a/docs/content/cdc-ingestion/pulsar-cdc.md b/docs/docs/cdc-ingestion/pulsar-cdc.md
similarity index 91%
rename from docs/content/cdc-ingestion/pulsar-cdc.md
rename to docs/docs/cdc-ingestion/pulsar-cdc.md
index 2ac1e7466d..048be02d3a 100644
--- a/docs/content/cdc-ingestion/pulsar-cdc.md
+++ b/docs/docs/cdc-ingestion/pulsar-cdc.md
@@ -1,10 +1,16 @@
---
title: "Pulsar CDC"
-weight: 5
-type: docs
-aliases:
-- /cdc-ingestion/pulsar-cdc.html
+sidebar_position: 5
---
+
+```mdx-code-block
+import ConfigTable from '@site/src/components/ConfigTable';
+import pulsarSyncTableHtml from '@site/generated/pulsar_sync_table.html';
+import pulsarSyncDatabaseHtml from '@site/generated/pulsar_sync_database.html';
+```
+
+
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -66,7 +72,8 @@ If a message in a pulsar topic is a change event captured from another database
</tbody>
</table>
-{{< hint info >}}
+:::info
+
The JSON sources possibly missing some information. For example, Ogg and Maxwell format standards don't contain field
types; When you write JSON sources into Flink Pulsar sink, it will only reserve data and row type and drop other information.
The synchronization job will try best to handle the problem as follows:
@@ -74,17 +81,18 @@ The synchronization job will try best to handle the problem as follows:
2. If missing database name or table name, you cannot do database synchronization, but you can still do table synchronization.
3. If missing primary keys, the job might create non primary key table. You can set primary keys when submit job in table
synchronization.
-{{< /hint >}}
+
+:::
## Synchronizing Tables
-By using [PulsarSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/pulsar/PulsarSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Pulsar's one topic into one Paimon table.
+By using [PulsarSyncTableAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/pulsar/PulsarSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Pulsar's one topic into one Paimon table.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
pulsar_sync_table \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -98,7 +106,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/pulsar_sync_table >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={pulsarSyncTableHtml} />
+```
+```
If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified Pulsar topic's tables,it gets the earliest non-DDL data parsing schema from topic. If the Paimon table already exists, its schema will be compared against the schema of all specified Pulsar topic's tables.
@@ -106,7 +118,7 @@ Example 1:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
pulsar_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -151,7 +163,7 @@ Then you can submit synchronization job:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
pulsar_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -165,7 +177,7 @@ For some append data (such as log data), it can be treated as special CDC data w
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
kafka_sync_table \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -183,13 +195,13 @@ For some append data (such as log data), it can be treated as special CDC data w
## Synchronizing Databases
-By using [PulsarSyncDatabaseAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/pulsar/PulsarSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the multi topic or one topic into one Paimon database.
+By using [PulsarSyncDatabaseAction](https://paimon.apache.org/docs/master/api/java/org/apache/paimon/flink/action/cdc/pulsar/PulsarSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the multi topic or one topic into one Paimon database.
To use this feature through `flink run`, run the following shell command.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
pulsar_sync_database \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -205,7 +217,11 @@ To use this feature through `flink run`, run the following shell command.
[--table_conf <paimon-table-sink-conf> [--table_conf <paimon-table-sink-conf> ...]]
```
-{{< generated/pulsar_sync_database >}}
+```mdx-code-block
+```mdx-code-block
+<ConfigTable html={pulsarSyncDatabaseHtml} />
+```
+```
Only tables with primary keys will be synchronized.
@@ -220,7 +236,7 @@ Synchronization from one Pulsar topic to Paimon database.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
pulsar_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
@@ -240,7 +256,7 @@ Synchronization from multiple Pulsar topics to Paimon database.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
pulsar_sync_database \
--warehouse hdfs:///path/to/warehouse \
--database test_db \
diff --git a/docs/content/concepts/basic-concepts.md b/docs/docs/concepts/basic-concepts.md
similarity index 82%
rename from docs/content/concepts/basic-concepts.md
rename to docs/docs/concepts/basic-concepts.md
index 8523d29f3f..0bbef8d4b0 100644
--- a/docs/content/concepts/basic-concepts.md
+++ b/docs/docs/concepts/basic-concepts.md
@@ -1,10 +1,8 @@
---
title: "Basic Concepts"
-weight: 2
-type: docs
-aliases:
-- /concepts/basic-concepts.html
+sidebar_position: 2
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,7 +28,7 @@ under the License.
All files of a table are stored under one base directory. Paimon files are organized in a layered style. The following image illustrates the file layout. Starting from a snapshot file, Paimon readers can recursively access all records from the table.
-{{< img src="/img/file-layout.png">}}
+
## Snapshot
@@ -67,9 +65,9 @@ By partitioning, users can efficiently operate on a slice of records in the tabl
## Consistency Guarantees
Paimon writers use two-phase commit protocol to atomically commit a batch of records to the table. Each commit produces
-at most two [snapshots]({{< ref "concepts/basic-concepts#snapshot" >}}) at commit time. It depends on the incremental write and compaction strategy. If only incremental writes are performed without triggering a compaction operation, only an incremental snapshot will be created. If a compaction operation is triggered, an incremental snapshot and a compacted snapshot will be created.
+at most two [snapshots](./basic-concepts#snapshot) at commit time. It depends on the incremental write and compaction strategy. If only incremental writes are performed without triggering a compaction operation, only an incremental snapshot will be created. If a compaction operation is triggered, an incremental snapshot and a compacted snapshot will be created.
For any two writers modifying a table at the same time, as long as they do not modify the same partition, their commits
can occur in parallel. If they modify the same partition, only snapshot isolation is guaranteed. That is, the final table
state may be a mix of the two commits, but no changes are lost.
-See [dedicated compaction job]({{< ref "maintenance/dedicated-compaction#dedicated-compaction-job" >}}) for more info.
+See [dedicated compaction job](../maintenance/dedicated-compaction#dedicated-compaction-job) for more info.
diff --git a/docs/content/concepts/catalog.md b/docs/docs/concepts/catalog.md
similarity index 96%
rename from docs/content/concepts/catalog.md
rename to docs/docs/concepts/catalog.md
index 3d57cbf129..7d16131e1b 100644
--- a/docs/content/concepts/catalog.md
+++ b/docs/docs/concepts/catalog.md
@@ -1,10 +1,8 @@
---
title: "Catalog"
-weight: 4
-type: docs
-aliases:
-- /concepts/catalog.html
+sidebar_position: 4
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -54,7 +52,7 @@ CREATE CATALOG my_catalog WITH (
## REST Catalog
By using the Paimon REST catalog, changes to the catalog will be directly stored in a remote catalog server which exposed through REST API.
-See [Paimon REST Catalog]({{< ref "concepts/rest/overview" >}}).
+See [Paimon REST Catalog](./rest/overview).
## Hive Catalog
diff --git a/docs/content/concepts/concurrency-control.md b/docs/docs/concepts/concurrency-control.md
similarity index 91%
rename from docs/content/concepts/concurrency-control.md
rename to docs/docs/concepts/concurrency-control.md
index 71a7c31dc8..261d563535 100644
--- a/docs/content/concepts/concurrency-control.md
+++ b/docs/docs/concepts/concurrency-control.md
@@ -1,10 +1,8 @@
---
title: "Concurrency Control"
-weight: 3
-type: docs
-aliases:
-- /concepts/concurrency-control.html
+sidebar_position: 3
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -39,7 +37,7 @@ There may be two types of commit failures here:
Paimon's snapshot ID is unique, so as long as the job writes its snapshot file to the file system, it is considered successful.
-{{< img src="/img/snapshot-conflict.png">}}
+
Paimon uses the file system's renaming mechanism to commit snapshots, which is secure for HDFS as it ensures
transactional and atomic renaming.
@@ -54,7 +52,7 @@ If there are conflicts (which means the file has been logically deleted), it can
so it can only intentionally trigger a failover to restart, and the job will retrieve the latest status from the filesystem
in the hope of resolving this conflict.
-{{< img src="/img/files-conflict.png">}}
+
Paimon will ensure that there is no data loss or duplication here, but if two streaming jobs are writing at the same
time and there are conflicts, you will see that they are constantly restarting, which is not a good thing.
@@ -63,4 +61,4 @@ The essence of conflict lies in deleting files (logically), and deleting files i
we close the compaction of the writing job (Set 'write-only' to true) and start a separate job to do the compaction work,
everything is very good.
-See [dedicated compaction job]({{< ref "maintenance/dedicated-compaction#dedicated-compaction-job" >}}) for more info.
+See [dedicated compaction job](../maintenance/dedicated-compaction#dedicated-compaction-job) for more info.
diff --git a/docs/content/concepts/data-types.md b/docs/docs/concepts/data-types.md
similarity index 98%
rename from docs/content/concepts/data-types.md
rename to docs/docs/concepts/data-types.md
index acf6a3c0ef..39e367b46a 100644
--- a/docs/content/concepts/data-types.md
+++ b/docs/docs/concepts/data-types.md
@@ -1,10 +1,8 @@
---
title: "Data Types"
-weight: 8
-type: docs
-aliases:
-- /concepts/data-types.html
+sidebar_position: 8
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -186,7 +184,7 @@ All data types supported by Paimon are as follows:
<td><code>BLOB</code></td>
<td><code>Data type of a binary large object.</code><br><br>
<code>Designed for storing large binary data such as images, videos, audio files, and other multimodal data. Unlike BYTES type which stores data inline, BLOB stores large binary data in separate files and maintains references to them, providing better performance for large objects.</code><br><br>
- <code>Note: Requires 'row-tracking.enabled' and 'data-evolution.enabled' to be set to true. See <a href="{{< ref "append-table/blob" >}}">Blob Type</a> for details.</code>
+ <code>Note: Requires 'row-tracking.enabled' and 'data-evolution.enabled' to be set to true. See <a href="../append-table/blob">Blob Type</a> for details.</code>
</td>
</tr>
</tbody>
diff --git a/docs/content/concepts/functions.md b/docs/docs/concepts/functions.md
similarity index 95%
rename from docs/content/concepts/functions.md
rename to docs/docs/concepts/functions.md
index 4430e37c3a..1c6788d513 100644
--- a/docs/content/concepts/functions.md
+++ b/docs/docs/concepts/functions.md
@@ -1,10 +1,8 @@
---
title: "Functions"
-weight: 9
-type: docs
-aliases:
-- /concepts/functions.html
+sidebar_position: 9
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -86,4 +84,4 @@ This statement deletes the existing `parse_str` function from the `mydb` databas
## Functions in Spark
-see [SQL Functions]({{< ref "spark/sql-functions#user-defined-function" >}})
+see [SQL Functions](../spark/sql-functions#user-defined-function)
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/concepts/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/concepts/index.md
index eacaa231ca..e550041e2d 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/concepts/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Concepts"
+sidebar_position: 1
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/overview.md b/docs/docs/concepts/overview.md
similarity index 96%
rename from docs/content/concepts/overview.md
rename to docs/docs/concepts/overview.md
index 28fab2017e..8eaa6f558d 100644
--- a/docs/content/concepts/overview.md
+++ b/docs/docs/concepts/overview.md
@@ -1,10 +1,8 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /concepts/overview.html
+sidebar_position: 1
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -28,7 +26,7 @@ under the License.
Apache Paimon's Architecture:
-{{< img src="/img/architecture.png">}}
+
As shown in the architecture above:
diff --git a/docs/content/concepts/rest/bear.md b/docs/docs/concepts/rest/bear.md
similarity index 96%
rename from docs/content/concepts/rest/bear.md
rename to docs/docs/concepts/rest/bear.md
index 0a54271837..86b5570a05 100644
--- a/docs/content/concepts/rest/bear.md
+++ b/docs/docs/concepts/rest/bear.md
@@ -1,10 +1,8 @@
---
title: "Bear Token"
-weight: 2
-type: docs
-aliases:
-- /concepts/rest/bear.html
+sidebar_position: 2
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/rest/dlf.md b/docs/docs/concepts/rest/dlf.md
similarity index 98%
rename from docs/content/concepts/rest/dlf.md
rename to docs/docs/concepts/rest/dlf.md
index ce6b6fbe4d..2a8314a49a 100644
--- a/docs/content/concepts/rest/dlf.md
+++ b/docs/docs/concepts/rest/dlf.md
@@ -1,11 +1,9 @@
---
title: "DLF Token"
-weight: 3
-type: docs
-aliases:
- - /concepts/rest/dlf.html
+sidebar_position: 3
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -33,9 +31,11 @@ storage analysis, and storage optimization.
DLF provides multiple authentication methods for different environments.
-{{< hint info >}}
+:::info
+
The `'warehouse'` is your catalog instance name on the server, not the path.
-{{< /hint >}}
+
+:::
## Use the access key
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/concepts/rest/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/concepts/rest/index.md
index eacaa231ca..6d2ecc656b 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/concepts/rest/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "RESTCatalog"
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/rest/overview.md b/docs/docs/concepts/rest/overview.md
similarity index 84%
rename from docs/content/concepts/rest/overview.md
rename to docs/docs/concepts/rest/overview.md
index dee9a9c257..d73863a2d3 100644
--- a/docs/content/concepts/rest/overview.md
+++ b/docs/docs/concepts/rest/overview.md
@@ -1,10 +1,8 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /concepts/rest/overview.html
+sidebar_position: 1
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -29,9 +27,9 @@ under the License.
## Overview
Paimon REST Catalog provides a lightweight implementation to access the catalog service. Paimon could access the
-catalog service through a catalog server which implements REST API. You can see all APIs in [REST API]({{< ref "concepts/rest/rest-api" >}}).
+catalog service through a catalog server which implements REST API. You can see all APIs in [REST API](./rest-api).
-{{< img src="/img/rest-catalog.svg">}}
+
## Key Features
@@ -50,7 +48,7 @@ catalog service through a catalog server which implements REST API. You can see
## Conclusion
-REST Catalog offers adaptable solution for accessing the catalog service. According to [REST API]({{< ref "concepts/rest/rest-api" >}}) is decoupled
+REST Catalog offers adaptable solution for accessing the catalog service. According to [REST API](./rest-api) is decoupled
from the catalog service.
Technology-specific Logic is encapsulated on the catalog server. At the same time, the catalog server supports any
@@ -60,9 +58,9 @@ backend and languages.
RESTCatalog supports multiple access authentication methods, including the following:
-1. [Bear Token]({{< ref "concepts/rest/bear" >}}).
-2. [DLF Token]({{< ref "concepts/rest/dlf" >}}).
+1. [Bear Token](./bear).
+2. [DLF Token](./dlf).
## REST Open API
-See [REST API]({{< ref "concepts/rest/rest-api" >}}).
+See [REST API](./rest-api).
diff --git a/docs/content/concepts/rest/pvfs.md b/docs/docs/concepts/rest/pvfs.md
similarity index 98%
rename from docs/content/concepts/rest/pvfs.md
rename to docs/docs/concepts/rest/pvfs.md
index 7a68351458..772819488d 100644
--- a/docs/content/concepts/rest/pvfs.md
+++ b/docs/docs/concepts/rest/pvfs.md
@@ -1,10 +1,8 @@
---
title: "PVFS"
-weight: 6
-type: docs
-aliases:
- - /concepts/rest/pvfs.html
+sidebar_position: 6
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/rest/rest-api.md b/docs/docs/concepts/rest/rest-api.md
similarity index 87%
rename from docs/content/concepts/rest/rest-api.md
rename to docs/docs/concepts/rest/rest-api.md
index 6aa8175c7a..c22e3784cc 100644
--- a/docs/content/concepts/rest/rest-api.md
+++ b/docs/docs/concepts/rest/rest-api.md
@@ -1,9 +1,8 @@
---
title: "REST API"
-layout: "custom-page"
-bookToc: false
-redocPage: true
+hide_table_of_contents: true
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -24,5 +23,5 @@ under the License.
-->
<body>
- {{< redoc_rest_catalog_api >}}
+ <iframe src="/docs/master/rest-catalog-open-api.yaml" width="100%" height="800px" />
</body>
diff --git a/docs/content/concepts/rest/tables.md b/docs/docs/concepts/rest/tables.mdx
similarity index 80%
rename from docs/content/concepts/rest/tables.md
rename to docs/docs/concepts/rest/tables.mdx
index 4a9f8ff102..508b715f25 100644
--- a/docs/content/concepts/rest/tables.md
+++ b/docs/docs/concepts/rest/tables.mdx
@@ -1,10 +1,11 @@
---
title: "Tables"
-weight: 5
-type: docs
-aliases:
-- /concepts/rest/tables.html
+sidebar_position: 5
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -37,7 +38,7 @@ Paimon supports tables:
### Primary Key Table
-See [Paimon with Primary key]({{< ref "primary-key-table/overview" >}}).
+See [Paimon with Primary key](../../primary-key-table/overview).
Primary keys consist of a set of columns that contain unique values for each record. Paimon enforces data ordering by
sorting the primary key within each bucket, allowing streaming update and streaming changelog read.
@@ -45,8 +46,9 @@ sorting the primary key within each bucket, allowing streaming update and stream
The definition of primary key is similar to that of standard SQL, as it ensures that there is only one data entry for
the same primary key during batch queries.
-{{< tabs "primary-table" >}}
-{{< tab "Flink SQL" >}}
+<Tabs groupId="primary-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
```sql
CREATE TABLE my_table (
@@ -56,9 +58,10 @@ CREATE TABLE my_table (
'bucket'='8'
)
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
```sql
CREATE TABLE my_table (
@@ -70,12 +73,13 @@ CREATE TABLE my_table (
)
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
### Append Table
-See [Append Table]({{< ref "append-table/overview" >}}).
+See [Append Table](../../append-table/overview).
If a table does not have a primary key defined, it is an append table. Compared to the primary key table, it does not
have the ability to directly receive changelogs. It cannot be directly updated with data through streaming upsert. It
@@ -103,8 +107,9 @@ directory structure.
Currently only support `CSV`, `Parquet`, `ORC`, `JSON` formats.
-{{< tabs "format-table" >}}
-{{< tab "Flink-CSV" >}}
+<Tabs groupId="format-table">
+
+<TabItem value="flink-csv" label="Flink-CSV">
```sql
CREATE TABLE my_csv_table (
@@ -116,9 +121,10 @@ CREATE TABLE my_csv_table (
'csv.field-delimiter'=','
)
```
-{{< /tab >}}
-{{< tab "Spark-CSV" >}}
+</TabItem>
+
+<TabItem value="spark-csv" label="Spark-CSV">
```sql
CREATE TABLE my_csv_table (
@@ -127,9 +133,9 @@ CREATE TABLE my_csv_table (
) USING csv OPTIONS ('csv.field-delimiter' ',')
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink-Parquet" >}}
+<TabItem value="flink-parquet" label="Flink-Parquet">
```sql
CREATE TABLE my_parquet_table (
@@ -140,9 +146,10 @@ CREATE TABLE my_parquet_table (
'file.format'='parquet'
)
```
-{{< /tab >}}
-{{< tab "Spark-Parquet" >}}
+</TabItem>
+
+<TabItem value="spark-parquet" label="Spark-Parquet">
```sql
CREATE TABLE my_parquet_table (
@@ -151,9 +158,9 @@ CREATE TABLE my_parquet_table (
) USING parquet
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink-JSON" >}}
+<TabItem value="flink-json" label="Flink-JSON">
```sql
CREATE TABLE my_json_table (
@@ -164,9 +171,10 @@ CREATE TABLE my_json_table (
'file.format'='json'
)
```
-{{< /tab >}}
-{{< tab "Spark-JSON" >}}
+</TabItem>
+
+<TabItem value="spark-json" label="Spark-JSON">
```sql
CREATE TABLE my_json_table (
@@ -175,9 +183,9 @@ CREATE TABLE my_json_table (
) USING json
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Object Table
@@ -191,22 +199,29 @@ Table.
To Create an object table:
-{{< tabs "create-object-table" >}}
-{{< tab "Flink-SQL" >}}
+<Tabs groupId="create-object-table">
+
+<TabItem value="flink-sql" label="Flink-SQL">
+
```sql
CREATE TABLE `my_object_table` WITH (
'type' = 'object-table'
);
```
-{{< /tab >}}
-{{< tab "Spark-SQL" >}}
+
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark-SQL">
+
```sql
CREATE TABLE `my_object_table` TBLPROPERTIES (
'type' = 'object-table'
);
```
-{{< /tab >}}
-{{< /tabs >}}
-We recommend using [pvfs]({{< ref "concepts/rest/pvfs" >}}). to access files in the object table, access to the files
+</TabItem>
+
+</Tabs>
+
+We recommend using [pvfs](./pvfs). to access files in the object table, access to the files
through the permission system of Paimon REST Catalog.
diff --git a/docs/content/concepts/spec/datafile.md b/docs/docs/concepts/spec/datafile.md
similarity index 98%
rename from docs/content/concepts/spec/datafile.md
rename to docs/docs/concepts/spec/datafile.md
index 75df2f588d..8e24662519 100644
--- a/docs/content/concepts/spec/datafile.md
+++ b/docs/docs/concepts/spec/datafile.md
@@ -1,10 +1,8 @@
---
title: "DataFile"
-weight: 5
-type: docs
-aliases:
-- /concepts/spec/datafile.html
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/spec/fileformat.md b/docs/docs/concepts/spec/fileformat.md
similarity index 99%
rename from docs/content/concepts/spec/fileformat.md
rename to docs/docs/concepts/spec/fileformat.md
index 27d4f63d81..baac8f2b05 100644
--- a/docs/content/concepts/spec/fileformat.md
+++ b/docs/docs/concepts/spec/fileformat.md
@@ -1,10 +1,8 @@
---
title: "FileFormat"
-weight: 7
-type: docs
-aliases:
-- /concepts/spec/fileformat.html
+sidebar_position: 7
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -141,7 +139,7 @@ The following table lists the type mapping from Paimon type to Parquet type.
</table>
Limitations:
-1. [Parquet does not support nullable map keys](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps).
+1. [Parquet does not support nullable map keys](https://github.com/apache/parquet-format/blob/master/LogicalTypes#maps).
2. Parquet TIMESTAMP type with precision 9 will use INT96, but this int96 is a time zone converted value and requires additional adjustments.
## AVRO
@@ -790,4 +788,4 @@ Limitations:
2. BLOB format does not support predicate pushdown.
3. Statistics collection is not supported for BLOB columns.
-For usage details, configuration options, and examples, see [Blob Type]({{< ref "append-table/blob" >}}).
+For usage details, configuration options, and examples, see [Blob Type](../../append-table/blob).
diff --git a/docs/content/concepts/spec/fileindex.md b/docs/docs/concepts/spec/fileindex.mdx
similarity index 98%
rename from docs/content/concepts/spec/fileindex.md
rename to docs/docs/concepts/spec/fileindex.mdx
index 29402258c9..e2f0899f7d 100644
--- a/docs/content/concepts/spec/fileindex.md
+++ b/docs/docs/concepts/spec/fileindex.mdx
@@ -1,10 +1,11 @@
---
title: "File Index"
-weight: 8
-type: docs
-aliases:
-- /concepts/spec/fileindex.html
+sidebar_position: 8
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -102,9 +103,9 @@ This class use (64-bits) long hash. Store the num hash function (one integer) an
* `file-index.bitmap.columns`: specify the columns that need bitmap index.
* `file-index.bitmap.<column_name>.index-block-size`: to config secondary index block size, default value is 16kb.
-{{< tabs "bitmap" >}}
+<Tabs groupId="bitmap">
-{{< tab "V2" >}}
+<TabItem value="v2" label="V2">
Bitmap file index format (V2):
@@ -167,9 +168,9 @@ length: 4 bytes int
</pre>
-{{< /tab >}}
+</TabItem>
-{{< tab "V1 (Legacy)" >}}
+<TabItem value="v1-legacy" label="V1 (Legacy)">
(Legacy) Bitmap file index format (V1):
@@ -211,9 +212,9 @@ offset: 4 bytes int (when it is negative, it represents t
and its position is the inverse of the negative value)
</pre>
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
Integers are all BIG_ENDIAN.
@@ -348,11 +349,11 @@ RangeBitmap only support the following data type: TinyIntType, SmallIntType, Int
## Index: Bit-Slice Index Bitmap
-{{< hint warning >}}
+:::warning
Deprecated. Using the range-bitmap index instead.
-{{< /hint >}}
+:::
BSI file index is a numeric range index, used to accelerate range query, it can be used with bitmap index.
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/concepts/spec/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/concepts/spec/index.md
index eacaa231ca..746a635d1f 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/concepts/spec/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Specification"
+sidebar_position: 11
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/spec/manifest.md b/docs/docs/concepts/spec/manifest.md
similarity index 97%
rename from docs/content/concepts/spec/manifest.md
rename to docs/docs/concepts/spec/manifest.md
index 39d3dc7724..adaa6e9106 100644
--- a/docs/content/concepts/spec/manifest.md
+++ b/docs/docs/concepts/spec/manifest.md
@@ -1,10 +1,8 @@
---
title: "Manifest"
-weight: 4
-type: docs
-aliases:
-- /concepts/spec/manifest.html
+sidebar_position: 4
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -92,7 +90,7 @@ The data file meta is:
### Index Manifest
-Index Manifest includes meta of several [table-index]({{< ref "concepts/spec/tableindex" >}}) files.
+Index Manifest includes meta of several [table-index](./tableindex) files.
```shell
├── manifest
diff --git a/docs/content/concepts/spec/overview.md b/docs/docs/concepts/spec/overview.md
similarity index 96%
rename from docs/content/concepts/spec/overview.md
rename to docs/docs/concepts/spec/overview.md
index a48e8451e3..346e02759b 100644
--- a/docs/content/concepts/spec/overview.md
+++ b/docs/docs/concepts/spec/overview.md
@@ -1,10 +1,8 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /concepts/spec/overview.html
+sidebar_position: 1
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -29,7 +27,7 @@ under the License.
This is the specification for the Paimon table format, this document standardizes the underlying file structure and
design of Paimon.
-{{< img src="/img/file-layout.png">}}
+
## Terms
diff --git a/docs/content/concepts/spec/schema.md b/docs/docs/concepts/spec/schema.md
similarity index 98%
rename from docs/content/concepts/spec/schema.md
rename to docs/docs/concepts/spec/schema.md
index 60985cc036..0f408a82d2 100644
--- a/docs/content/concepts/spec/schema.md
+++ b/docs/docs/concepts/spec/schema.md
@@ -1,10 +1,8 @@
---
title: "Schema"
-weight: 2
-type: docs
-aliases:
-- /concepts/spec/schema.html
+sidebar_position: 2
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/spec/snapshot.md b/docs/docs/concepts/spec/snapshot.md
similarity index 97%
rename from docs/content/concepts/spec/snapshot.md
rename to docs/docs/concepts/spec/snapshot.md
index f000a07da1..f9762b0599 100644
--- a/docs/content/concepts/spec/snapshot.md
+++ b/docs/docs/concepts/spec/snapshot.md
@@ -1,10 +1,8 @@
---
title: "Snapshot"
-weight: 3
-type: docs
-aliases:
-- /concepts/spec/snapshot.html
+sidebar_position: 3
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/spec/tableindex.md b/docs/docs/concepts/spec/tableindex.md
similarity index 96%
rename from docs/content/concepts/spec/tableindex.md
rename to docs/docs/concepts/spec/tableindex.md
index 6886fc6fef..156b603ef0 100644
--- a/docs/content/concepts/spec/tableindex.md
+++ b/docs/docs/concepts/spec/tableindex.md
@@ -1,10 +1,8 @@
---
title: "Table Index"
-weight: 7
-type: docs
-aliases:
-- /concepts/spec/tableindex.html
+sidebar_position: 7
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -43,7 +41,7 @@ HASH_VALUE is the hash value of the primary-key. 4 bytes, BIG_ENDIAN.
Deletion file is used to store the deleted records position for each data file. Each bucket has one deletion file for
primary key table.
-{{< img src="/img/deletion-file.png">}}
+
The deletion file is a binary file, and the format is as follows:
diff --git a/docs/content/concepts/system-tables.md b/docs/docs/concepts/system-tables.mdx
similarity index 98%
rename from docs/content/concepts/system-tables.md
rename to docs/docs/concepts/system-tables.mdx
index 2eb2fc73c3..59ca574244 100644
--- a/docs/content/concepts/system-tables.md
+++ b/docs/docs/concepts/system-tables.mdx
@@ -1,10 +1,11 @@
---
title: "System Tables"
-weight: 7
-type: docs
-aliases:
-- /concepts/system-tables.html
+sidebar_position: 7
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -92,7 +93,7 @@ SELECT s.snapshot_id, t.schema_id, t.fields
### Options Table
-You can query the table's option information which is specified from the DDL through options table. The options not shown will be the default value. You can take reference to [Configuration]({{< ref "maintenance/configurations#coreoptions" >}}).
+You can query the table's option information which is specified from the DDL through options table. The options not shown will be the default value. You can take reference to [Configuration](../maintenance/configurations#coreoptions).
```sql
SELECT * FROM my_table$options;
@@ -138,15 +139,18 @@ SELECT * FROM my_table$audit_log;
For primary key tables, you can enable the `table-read.sequence-number.enabled` option to include the `_SEQUENCE_NUMBER` field in the output.
-{{< tabs "audit-log-sequence-number" >}}
+<Tabs groupId="audit-log-sequence-number">
+
+<TabItem value="enable-via-alter-table" label="Enable via ALTER TABLE">
-{{< tab "Enable via ALTER TABLE" >}}
```sql
ALTER TABLE my_table SET ('table-read.sequence-number.enabled' = 'true');
```
-{{< /tab >}}
-{{< tab "Enable via CREATE TABLE" >}}
+</TabItem>
+
+<TabItem value="enable-via-create-table" label="Enable via CREATE TABLE">
+
```sql
CREATE TABLE my_table (
...
@@ -154,9 +158,10 @@ CREATE TABLE my_table (
'table-read.sequence-number.enabled' = 'true'
);
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
```sql
SELECT * FROM my_table$audit_log;
@@ -175,9 +180,11 @@ SELECT * FROM my_table$audit_log;
*/
```
-{{< hint info >}}
+:::info
+
The `table-read.sequence-number.enabled` option cannot be set via SQL hints.
-{{< /hint >}}
+
+:::
### Binlog Table
@@ -228,12 +235,12 @@ Read-optimized system table improves reading performance by only scanning files
For primary-key tables, `ro` system table only scans files on the topmost level.
That is to say, `ro` system table only produces the result of the latest full compaction.
-{{< hint info >}}
+:::info
It is possible that different buckets carry out full compaction at difference times,
so it is possible that the values of different keys come from different snapshots.
-{{< /hint >}}
+:::
For append tables, as all files can be read without merging,
`ro` system table acts like the normal append table.
@@ -548,7 +555,7 @@ This table also displays various statistics information of partition.
### ALL Options Table
-This table is similar to [Options Table]({{< ref "#options-table" >}}), but it shows all the table options in all database.
+This table is similar to [Options Table](#options-table), but it shows all the table options in all database.
```sql
SELECT * FROM sys.all_table_options;
@@ -568,7 +575,7 @@ SELECT * FROM sys.all_table_options;
### Catalog Options Table
-You can query the catalog's option information through catalog options table. The options not shown will be the default value. You can take reference to [Configuration]({{< ref "maintenance/configurations#coreoptions" >}}).
+You can query the catalog's option information through catalog options table. The options not shown will be the default value. You can take reference to [Configuration](../maintenance/configurations#coreoptions).
```sql
SELECT * FROM sys.catalog_options;
diff --git a/docs/content/concepts/views.md b/docs/docs/concepts/views.md
similarity index 93%
rename from docs/content/concepts/views.md
rename to docs/docs/concepts/views.md
index f5ad50edf9..171863ae65 100644
--- a/docs/content/concepts/views.md
+++ b/docs/docs/concepts/views.md
@@ -1,10 +1,8 @@
---
title: "Views"
-weight: 10
-type: docs
-aliases:
-- /concepts/views.html
+sidebar_position: 10
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -103,6 +101,6 @@ CALL sys.alter_view_dialect('view_identifier', 'drop', 'spark');
## See also
-- [Spark SQL DDL – Views]({{< ref "spark/sql-ddl#view" >}})
-- [REST Catalog Overview]({{< ref "concepts/rest/overview" >}})
-- [REST Catalog View API]({{< ref "concepts/rest/rest-api" >}})
+- [Spark SQL DDL – Views](../spark/sql-ddl#view)
+- [REST Catalog Overview](./rest/overview)
+- [REST Catalog View API](./rest/rest-api)
diff --git a/docs/content/ecosystem/amoro.md b/docs/docs/ecosystem/amoro.md
similarity index 95%
rename from docs/content/ecosystem/amoro.md
rename to docs/docs/ecosystem/amoro.md
index 05c0b90fc8..e9c9d3d600 100644
--- a/docs/content/ecosystem/amoro.md
+++ b/docs/docs/ecosystem/amoro.md
@@ -1,10 +1,8 @@
---
title: "Amoro"
-weight: 6
-type: docs
-aliases:
-- /ecosystem/amoro.html
+sidebar_position: 6
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -36,7 +34,7 @@ under the License.
Apache Amoro supports all catalog types supported by paimon, including common catalog: Hadoop, Hive, Glue, JDBC, Nessie and other third-party catalog.
Amoro supports all storage types supported by Paimon, including common store: Hadoop, S3, GCS, ECS, OSS, and so on.
-{{< img src="/img/amoro-paimon.png">}}
+
In the future, Paimon automatic optimization strategy will be supported, and users can achieve the best balance experience by cooperating with Amoro automatic optimization
diff --git a/docs/content/ecosystem/doris.md b/docs/docs/ecosystem/doris.md
similarity index 99%
rename from docs/content/ecosystem/doris.md
rename to docs/docs/ecosystem/doris.md
index 2db77bd7d8..878bba7257 100644
--- a/docs/content/ecosystem/doris.md
+++ b/docs/docs/ecosystem/doris.md
@@ -1,10 +1,8 @@
---
title: "Doris"
-weight: 3
-type: docs
-aliases:
-- /ecosystem/doris.html
+sidebar_position: 3
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/ecosystem/hive.md b/docs/docs/ecosystem/hive.mdx
similarity index 71%
rename from docs/content/ecosystem/hive.md
rename to docs/docs/ecosystem/hive.mdx
index f9c8b066b3..da316cc778 100644
--- a/docs/content/ecosystem/hive.md
+++ b/docs/docs/ecosystem/hive.mdx
@@ -1,10 +1,11 @@
---
title: "Hive"
-weight: 4
-type: docs
-aliases:
-- /ecosystem/hive.html
+sidebar_position: 4
---
+
+import Stable from '@site/src/components/Stable';
+import Unstable from '@site/src/components/Unstable';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -41,43 +42,51 @@ Note If you use beeline, please restart the hive cluster.
Download the jar file with corresponding version.
-{{< stable >}}
+<Stable>
+
+
| | Jar |
|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Hive 3.1 | [paimon-hive-connector-3.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/{{< version >}}/paimon-hive-connector-3.1-{{< version >}}.jar) |
-| Hive 2.3 | [paimon-hive-connector-2.3-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/{{< version >}}/paimon-hive-connector-2.3-{{< version >}}.jar) |
-| Hive 2.2 | [paimon-hive-connector-2.2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/{{< version >}}/paimon-hive-connector-2.2-{{< version >}}.jar) |
-| Hive 2.1 | [paimon-hive-connector-2.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/{{< version >}}/paimon-hive-connector-2.1-{{< version >}}.jar) |
-| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/{{< version >}}/paimon-hive-connector-2.1-cdh-6.3-{{< version >}}.jar) |
+| Hive 3.1 | [paimon-hive-connector-3.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/paimon-hive-connector-3.1-@@VERSION@@.jar) |
+| Hive 2.3 | [paimon-hive-connector-2.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/paimon-hive-connector-2.3-@@VERSION@@.jar) |
+| Hive 2.2 | [paimon-hive-connector-2.2-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/paimon-hive-connector-2.2-@@VERSION@@.jar) |
+| Hive 2.1 | [paimon-hive-connector-2.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/paimon-hive-connector-2.1-@@VERSION@@.jar) |
+| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar) |
+
+
+
+</Stable>
+
+<Unstable>
-{{< /stable >}}
-{{< unstable >}}
| | Jar |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Hive 3.1 | [paimon-hive-connector-3.1-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/{{< version >}}/) |
-| Hive 2.3 | [paimon-hive-connector-2.3-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/{{< version >}}/) |
-| Hive 2.2 | [paimon-hive-connector-2.2-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/{{< version >}}/) |
-| Hive 2.1 | [paimon-hive-connector-2.1-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/{{< version >}}/) |
-| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/{{< version >}}/) |
+| Hive 3.1 | [paimon-hive-connector-3.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/@@VERSION@@/) |
+| Hive 2.3 | [paimon-hive-connector-2.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.3/@@VERSION@@/) |
+| Hive 2.2 | [paimon-hive-connector-2.2-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.2/@@VERSION@@/) |
+| Hive 2.1 | [paimon-hive-connector-2.1-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1/@@VERSION@@/) |
+| Hive 2.1-cdh-6.3 | [paimon-hive-connector-2.1-cdh-6.3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-2.1-cdh-6.3/@@VERSION@@/) |
+
+
-{{< /unstable >}}
+</Unstable>
You can also manually build bundled jar from the source code.
-To build from source code, [clone the git repository]({{< github_repo >}}).
+To build from source code, [clone the git repository](@@GITHUB_REPO@@).
Build bundled jar with the following command.
`mvn clean install -DskipTests`
-You can find Hive connector jar in `./paimon-hive/paimon-hive-connector-<hive-version>/target/paimon-hive-connector-<hive-version>-{{< version >}}.jar`.
+You can find Hive connector jar in `./paimon-hive/paimon-hive-connector-<hive-version>/target/paimon-hive-connector-<hive-version>-@@VERSION@@.jar`.
There are several ways to add this jar to Hive.
-* You can create an `auxlib` folder under the root directory of Hive, and copy `paimon-hive-connector-{{< version >}}.jar` into `auxlib`.
-* You can also copy this jar to a path accessible by Hive, then use `add jar /path/to/paimon-hive-connector-{{< version >}}.jar` to enable paimon support in Hive. Note that this method is not recommended. If you're using the MR execution engine and running a join statement, you may be faced with the exception `org.apache.hive.com.esotericsoftware.kryo.kryoexception: unable to find class`.
+* You can create an `auxlib` folder under the root directory of Hive, and copy `paimon-hive-connector-@@VERSION@@.jar` into `auxlib`.
+* You can also copy this jar to a path accessible by Hive, then use `add jar /path/to/paimon-hive-connector-@@VERSION@@.jar` to enable paimon support in Hive. Note that this method is not recommended. If you're using the MR execution engine and running a join statement, you may be faced with the exception `org.apache.hive.com.esotericsoftware.kryo.kryoexception: unable to find class`.
NOTE:
@@ -91,7 +100,7 @@ NOTE:
Run the following Hive SQL in Hive CLI to access the created table.
```sql
--- Assume that paimon-hive-connector-<hive-version>-{{< version >}}.jar is already in auxlib directory.
+-- Assume that paimon-hive-connector-<hive-version>-@@VERSION@@.jar is already in auxlib directory.
-- List tables in Hive
-- (you might need to switch to "default" database if you're not there by default)
@@ -148,7 +157,7 @@ SET paimon.scan.snapshot-id=null;
You can create new paimon tables in Hive. Run the following Hive SQL in Hive CLI.
```sql
--- Assume that paimon-hive-connector-{{< version >}}.jar is already in auxlib directory.
+-- Assume that paimon-hive-connector-@@VERSION@@.jar is already in auxlib directory.
-- Let's create a new paimon table.
SET hive.metastore.warehouse.dir=warehouse_path;
@@ -165,7 +174,7 @@ STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';
To access existing paimon table, you can also register them as external tables in Hive. Run the following Hive SQL in Hive CLI.
```sql
--- Assume that paimon-hive-connector-{{< version >}}.jar is already in auxlib directory.
+-- Assume that paimon-hive-connector-@@VERSION@@.jar is already in auxlib directory.
-- Let's use the test_table created in the above section.
-- To create an external table, you don't need to specify any column or table properties.
-- Pointing the location to the path of table is enough.
@@ -214,12 +223,12 @@ OK
This section lists all supported type conversion between Hive and Paimon.
All Hive's data types are available in package `org.apache.hadoop.hive.serde2.typeinfo`.
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 10%">Hive Data Type</th>
- <th class="text-left" style="width: 10%">Paimon Data Type</th>
- <th class="text-left" style="width: 5%">Atomic Type</th>
+ <th className="text-left" style={{width: "10%"}}>Hive Data Type</th>
+ <th className="text-left" style={{width: "10%"}}>Paimon Data Type</th>
+ <th className="text-left" style={{width: "5%"}}>Atomic Type</th>
</tr>
</thead>
<tbody>
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/ecosystem/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/ecosystem/index.md
index eacaa231ca..7962021125 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/ecosystem/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Ecosystem"
+sidebar_position: 90
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/ecosystem/overview.md b/docs/docs/ecosystem/overview.md
similarity index 97%
rename from docs/content/ecosystem/overview.md
rename to docs/docs/ecosystem/overview.md
index eb28803ed7..ada5e1f448 100644
--- a/docs/content/ecosystem/overview.md
+++ b/docs/docs/ecosystem/overview.md
@@ -1,10 +1,8 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /ecosystem/overview.html
+sidebar_position: 1
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -78,4 +76,4 @@ You can also use Doris and Trino and Presto, or, you can just use Spark, Flink a
## Download
-[Download Link]({{< ref "project/download#engine-jars" >}})
+[Download Link](../project/download#engine-jars)
diff --git a/docs/content/ecosystem/starrocks.md b/docs/docs/ecosystem/starrocks.md
similarity index 98%
rename from docs/content/ecosystem/starrocks.md
rename to docs/docs/ecosystem/starrocks.md
index 49554d54aa..a155beaeea 100644
--- a/docs/content/ecosystem/starrocks.md
+++ b/docs/docs/ecosystem/starrocks.md
@@ -1,10 +1,8 @@
---
title: "StarRocks"
-weight: 2
-type: docs
-aliases:
-- /ecosystem/starrocks.html
+sidebar_position: 2
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/ecosystem/trino.md b/docs/docs/ecosystem/trino.md
similarity index 95%
rename from docs/content/ecosystem/trino.md
rename to docs/docs/ecosystem/trino.md
index 595f21b839..00eb9b72f9 100644
--- a/docs/content/ecosystem/trino.md
+++ b/docs/docs/ecosystem/trino.md
@@ -1,10 +1,8 @@
---
title: "Trino"
-weight: 5
-type: docs
-aliases:
-- /ecosystem/trino.html
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -40,11 +38,11 @@ filesystems for Trino on Trino official website.
## Preparing Paimon Jar File
-[Download]({{< ref "project/download" >}})
+[Download](../project/download)
You can also manually build a bundled jar from the source code. However, there are a few preliminary steps that need to be taken before compiling:
-- To build from the source code, [clone the git repository]({{< trino_github_repo >}}).
+- To build from the source code, [clone the git repository](@@TRINO_GITHUB_REPO@@).
- Install JDK21 locally, and configure JDK21 as a global environment variable;
Then,you can build bundled jar with the following command:
@@ -53,7 +51,7 @@ Then,you can build bundled jar with the following command:
mvn clean install -DskipTests
```
-You can find Trino connector jar in `./paimon-trino-<trino-version>/target/paimon-trino-<trino-version>-{{< version >}}-plugin.tar.gz`.
+You can find Trino connector jar in `./paimon-trino-<trino-version>/target/paimon-trino-<trino-version>-@@VERSION@@-plugin.tar.gz`.
We use [hadoop-apache](https://mvnrepository.com/artifact/io.trino.hadoop/hadoop-apache) as a dependency for Hadoop,
and the default Hadoop dependency typically supports both Hadoop 2 and Hadoop 3.
@@ -68,7 +66,7 @@ mvn clean install -DskipTests -Dhadoop.apache.version=3.3.5-1
### Install Paimon Connector
```bash
-tar -zxf paimon-trino-<trino-version>-{{< version >}}-plugin.tar.gz -C ${TRINO_HOME}/plugin
+tar -zxf paimon-trino-<trino-version>-@@VERSION@@-plugin.tar.gz -C ${TRINO_HOME}/plugin
```
> NOTE: For JDK 21, when Deploying Trino, should add jvm options: `--add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED`
@@ -95,7 +93,7 @@ For example, if you use oss as a storage, you can write in `paimon.properties` a
hive.config.resources=/path/to/core-site.xml
```
-Then, config core-site.xml according to [Jindo Reference](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto.md)
+Then, config core-site.xml according to [Jindo Reference](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto)
## Kerberos
@@ -174,11 +172,13 @@ SELECT * FROM t FOR VERSION AS OF 'my-tag';
```
-{{< hint warning >}}
+:::warning
+
If tag's name is a number and equals to a snapshot id, the VERSION AS OF syntax will consider tag first. For example, if
you have a tag named '1' based on snapshot 2, the statement `SELECT * FROM paimon.test_db.orders FOR VERSION AS OF '1'` actually queries snapshot 2
instead of snapshot 1.
-{{< /hint >}}
+
+:::
## Insert
diff --git a/docs/content/flink/action-jars.md b/docs/docs/flink/action-jars.md
similarity index 91%
rename from docs/content/flink/action-jars.md
rename to docs/docs/flink/action-jars.md
index 1d4269ca67..3c71dc727b 100644
--- a/docs/content/flink/action-jars.md
+++ b/docs/docs/flink/action-jars.md
@@ -1,10 +1,8 @@
---
title: "Action Jars"
-weight: 98
-type: docs
-aliases:
-- /flink/procedures.html
+sidebar_position: 98
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,7 +28,7 @@ After the Flink Local Cluster has been started, you can execute the action jar b
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
<action>
<args>
```
@@ -39,7 +37,7 @@ The following command is used to compact a table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--path <TABLE_PATH>
```
@@ -48,11 +46,13 @@ The following command is used to compact a table.
Paimon supports "MERGE INTO" via submitting the 'merge_into' job through `flink run`.
-{{< hint info >}}
+:::info
+
Important table properties setting:
-1. Only [primary key table]({{< ref "primary-key-table/overview" >}}) supports this feature.
+1. Only [primary key table](../primary-key-table/overview) supports this feature.
2. The action won't produce UPDATE_BEFORE, so it's not recommended to set 'changelog-producer' = 'input'.
- {{< /hint >}}
+
+:::
The design referenced such syntax:
```sql
@@ -79,7 +79,7 @@ Run the following command to submit a 'merge_into' job for the table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
merge_into \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -105,7 +105,7 @@ You can pass sqls by '--source_sql <sql> [, --source_sql <sql> ...]' to config e
-- Find all orders mentioned in the source table, then mark as important if the price is above 100
-- or delete if the price is under 10.
./flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
merge_into \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -121,7 +121,7 @@ You can pass sqls by '--source_sql <sql> [, --source_sql <sql> ...]' to config e
-- For matched order rows, increase the price, and if there is no match, insert the order from the
-- source table:
./flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
merge_into \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -136,7 +136,7 @@ You can pass sqls by '--source_sql <sql> [, --source_sql <sql> ...]' to config e
-- For not matched by source order rows (which are in the target table and does not match any row in the
-- source table based on the merge-condition), decrease the price or if the mark is 'trivial', delete them:
./flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
merge_into \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -152,7 +152,7 @@ You can pass sqls by '--source_sql <sql> [, --source_sql <sql> ...]' to config e
-- A --source_sql example:
-- Create a temporary view S in new catalog and use it as source table
./flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
merge_into \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -190,7 +190,8 @@ Parameters format:
4. not_matched_condition cannot use target table's columns to construct condition expression.
5. not_matched_by_source_condition cannot use source table's columns to construct condition expression.
-{{< hint warning >}}
+:::warning
+
1. Target alias cannot be duplicated with existed table name.
2. If the source table is not in the current catalog and current database, the source-table-name must be
qualified (database.table or catalog.database.table if created a new catalog).
@@ -218,13 +219,13 @@ Parameters format:
For example:\
\--source_sql "CREATE TABLE T (k INT) WITH ('special-key' = '123\\!')"
-{{< /hint >}}
+:::
For more information of 'merge_into', see
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
merge_into --help
```
@@ -236,7 +237,7 @@ Run the following command to submit a 'delete' job for the table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
delete \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -254,7 +255,7 @@ For more information of 'delete', see
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
delete --help
```
@@ -264,7 +265,7 @@ Run the following command to submit a 'drop_partition' job for the table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
drop_partition \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -280,7 +281,7 @@ For more information of 'drop_partition', see
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
drop_partition --help
```
@@ -290,7 +291,7 @@ Run the following command to submit a 'rewrite_file_index' job for the table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
rewrite_file_index \
--warehouse <warehouse-path> \
--identifier <database.table> \
@@ -302,7 +303,7 @@ For more information of 'rewrite_file_index', see
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
rewrite_file_index --help
```
@@ -314,7 +315,7 @@ submitting them as jobs. For example,
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
drop_partition \
--warehouse <warehouse-path> \
--database <database-name> \
diff --git a/docs/content/flink/consumer-id.md b/docs/docs/flink/consumer-id.mdx
similarity index 90%
rename from docs/content/flink/consumer-id.md
rename to docs/docs/flink/consumer-id.mdx
index fd10d9e813..cb855ca537 100644
--- a/docs/content/flink/consumer-id.md
+++ b/docs/docs/flink/consumer-id.mdx
@@ -1,10 +1,11 @@
---
title: "Consumer ID"
-weight: 5
-type: docs
-aliases:
-- /flink/consumer-id.html
+sidebar_position: 5
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -73,10 +74,12 @@ you can consider enabling `'consumer.mode' = 'at-least-once'` mode. This mode:
consumer. It doesn't affect the checkpoint time and have good performance.
2. This mode can provide more capabilities, such as watermark alignment.
-{{< hint >}}
+:::info
+
About `'consumer.mode'`, since the implementation of `exactly-once` mode and `at-least-once` mode are completely
different, the state of flink is incompatible and cannot be restored from the state when switching modes.
-{{< /hint >}}
+
+:::
## Reset Consumer
@@ -86,9 +89,9 @@ action job.
Run the following command:
-{{< tabs "reset_consumer" >}}
+<Tabs groupId="reset-consumer">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
```sql
CALL sys.reset_consumer(
@@ -98,13 +101,14 @@ CALL sys.reset_consumer(
);
-- No next_snapshot_id if you want to delete the consumer
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
reset-consumer \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -115,9 +119,10 @@ CALL sys.reset_consumer(
## No next_snapshot if you want to delete the consumer
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Clear Consumers
@@ -125,9 +130,9 @@ You can clear consumers in bulk with a given including consumers and excluding c
Run the following command:
-{{< tabs "clear_consumers" >}}
+<Tabs groupId="clear-consumers">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
```sql
CALL sys.clear_consumers(
@@ -137,13 +142,14 @@ CALL sys.clear_consumers(
);
-- No including_consumers if you want to clear all consumers except excluding_consumers in the table
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
clear_consumers \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -154,6 +160,7 @@ CALL sys.clear_consumers(
## No including_consumers if you want to clear all consumers except excluding_consumers in the table
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
diff --git a/docs/content/flink/default-value.md b/docs/docs/flink/default-value.md
similarity index 97%
rename from docs/content/flink/default-value.md
rename to docs/docs/flink/default-value.md
index 3da7ef5e7e..270abde3f6 100644
--- a/docs/content/flink/default-value.md
+++ b/docs/docs/flink/default-value.md
@@ -1,10 +1,8 @@
---
title: "Default Value"
-weight: 8
-type: docs
-aliases:
-- /flink/default-value.html
+sidebar_position: 8
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/flink/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/flink/index.md
index eacaa231ca..5ceaf76517 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/flink/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Engine Flink"
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/flink/procedures.md b/docs/docs/flink/procedures.md
similarity index 99%
rename from docs/content/flink/procedures.md
rename to docs/docs/flink/procedures.md
index 3202aedbbe..1e34926ae3 100644
--- a/docs/content/flink/procedures.md
+++ b/docs/docs/flink/procedures.md
@@ -1,10 +1,8 @@
---
title: "Procedures"
-weight: 97
-type: docs
-aliases:
-- /flink/procedures.html
+sidebar_position: 97
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/flink/quick-start.md b/docs/docs/flink/quick-start.mdx
similarity index 72%
rename from docs/content/flink/quick-start.md
rename to docs/docs/flink/quick-start.mdx
index 0317ce1bad..a9054a0673 100644
--- a/docs/content/flink/quick-start.md
+++ b/docs/docs/flink/quick-start.mdx
@@ -1,10 +1,13 @@
---
title: "Quick Start"
-weight: 1
-type: docs
-aliases:
-- /flink/quick-start.html
+sidebar_position: 1
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Stable from '@site/src/components/Stable';
+import Unstable from '@site/src/components/Unstable';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -35,44 +38,52 @@ Paimon currently supports Flink 2.2, 2.1, 2.0, 1.20, 1.19, 1.18, 1.17, 1.16. We
Download the jar file with corresponding version.
> Currently, paimon provides two types jar: one of which(the bundled jar) is used for read/write data, and the other(action jar) for operations such as manually compaction,
-{{< stable >}}
+<Stable>
+
+
| Version | Type | Jar |
|--------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Flink 2.2 | Bundled Jar | [paimon-flink-2.2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.2/{{< version >}}/paimon-flink-2.2-{{< version >}}.jar) |
-| Flink 2.1 | Bundled Jar | [paimon-flink-2.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.1/{{< version >}}/paimon-flink-2.1-{{< version >}}.jar) |
-| Flink 2.0 | Bundled Jar | [paimon-flink-2.0-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.0/{{< version >}}/paimon-flink-2.0-{{< version >}}.jar) |
-| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/{{< version >}}/paimon-flink-1.20-{{< version >}}.jar) |
-| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.19/{{< version >}}/paimon-flink-1.19-{{< version >}}.jar) |
-| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.18/{{< version >}}/paimon-flink-1.18-{{< version >}}.jar) |
-| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/{{< version >}}/paimon-flink-1.17-{{< version >}}.jar) |
-| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/{{< version >}}/paimon-flink-1.16-{{< version >}}.jar) |
-| Flink Action | Action Jar | [paimon-flink-action-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-action/{{< version >}}/paimon-flink-action-{{< version >}}.jar) |
+| Flink 2.2 | Bundled Jar | [paimon-flink-2.2-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.2/@@VERSION@@/paimon-flink-2.2-@@VERSION@@.jar) |
+| Flink 2.1 | Bundled Jar | [paimon-flink-2.1-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.1/@@VERSION@@/paimon-flink-2.1-@@VERSION@@.jar) |
+| Flink 2.0 | Bundled Jar | [paimon-flink-2.0-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.0/@@VERSION@@/paimon-flink-2.0-@@VERSION@@.jar) |
+| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/@@VERSION@@/paimon-flink-1.20-@@VERSION@@.jar) |
+| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.19/@@VERSION@@/paimon-flink-1.19-@@VERSION@@.jar) |
+| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.18/@@VERSION@@/paimon-flink-1.18-@@VERSION@@.jar) |
+| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/@@VERSION@@/paimon-flink-1.17-@@VERSION@@.jar) |
+| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/@@VERSION@@/paimon-flink-1.16-@@VERSION@@.jar) |
+| Flink Action | Action Jar | [paimon-flink-action-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-action/@@VERSION@@/paimon-flink-action-@@VERSION@@.jar) |
+
+
+
+</Stable>
+
+<Unstable>
-{{< /stable >}}
-{{< unstable >}}
| Version | Type | Jar |
|--------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| Flink 2.0 | Bundled Jar | Not yet released |
-| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.20/{{< version >}}/) |
-| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.19/{{< version >}}/) |
-| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.18/{{< version >}}/) |
-| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/{{< version >}}/) |
-| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/{{< version >}}/) |
-| Flink Action | Action Jar | [paimon-flink-action-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/{{< version >}}/) |
+| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.20/@@VERSION@@/) |
+| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.19/@@VERSION@@/) |
+| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.18/@@VERSION@@/) |
+| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/@@VERSION@@/) |
+| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/@@VERSION@@/) |
+| Flink Action | Action Jar | [paimon-flink-action-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/@@VERSION@@/) |
+
-{{< /unstable >}}
+
+</Unstable>
You can also manually build bundled jar from the source code.
-To build from source code, [clone the git repository]({{< github_repo >}}).
+To build from source code, [clone the git repository](@@GITHUB_REPO@@).
Build bundled jar with the following command.
- `mvn clean install -DskipTests`
-You can find the bundled jar in `./paimon-flink/paimon-flink-<flink-version>/target/paimon-flink-<flink-version>-{{< version >}}.jar`, and the action jar in `./paimon-flink/paimon-flink-action/target/paimon-flink-action-{{< version >}}.jar`.
+You can find the bundled jar in `./paimon-flink/paimon-flink-<flink-version>/target/paimon-flink-<flink-version>-@@VERSION@@.jar`, and the action jar in `./paimon-flink/paimon-flink-action/target/paimon-flink-action-@@VERSION@@.jar`.
## Start
@@ -94,9 +105,11 @@ cp paimon-flink-*.jar <FLINK_HOME>/lib/
**Step 3: Copy Hadoop Bundled Jar**
-{{< hint info >}}
+:::info
+
If the machine is in a hadoop environment, please ensure the value of the environment variable `HADOOP_CLASSPATH` include path to the common Hadoop libraries, you do not need to use the following pre-bundled Hadoop jar.
-{{< /hint >}}
+
+:::
[Download](https://flink.apache.org/downloads.html) Pre-bundled Hadoop jar and copy the jar file to the `lib` directory of your Flink home.
@@ -106,7 +119,7 @@ cp flink-shaded-hadoop-2-uber-*.jar <FLINK_HOME>/lib/
**Step 4: Start a Flink Local Cluster**
-In order to run multiple Flink jobs at the same time, you need to modify the cluster configuration in `<FLINK_HOME>/conf/flink-conf.yaml`(Flink version < 1.19) or `<FLINK_HOME>/conf/config.yaml`(Flink version >= 1.19).
+In order to run multiple Flink jobs at the same time, you need to modify the cluster configuration in `<FLINK_HOME>/conf/flink-conf.yaml`(Flink version \< 1.19) or `<FLINK_HOME>/conf/config.yaml`(Flink version >= 1.19).
```yaml
taskmanager.numberOfTaskSlots: 2
@@ -129,9 +142,9 @@ You can now start Flink SQL client to execute SQL scripts.
**Step 5: Create a Catalog and a Table**
-{{< tabs "Create Flink Catalog" >}}
+<Tabs groupId="create-flink-catalog">
-{{< tab "Catalog" >}}
+<TabItem value="catalog" label="Catalog">
```sql
-- if you're trying out Paimon in a distributed environment,
@@ -150,19 +163,21 @@ CREATE TABLE word_count (
);
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Generic-Catalog" >}}
+<TabItem value="generic-catalog" label="Generic-Catalog">
Using FlinkGenericCatalog, you need to use Hive metastore. Then, you can use all the tables from Paimon, Hive, and
Flink Generic Tables (Kafka and other tables)!
In this mode, you should use 'connector' option for creating tables.
-{{< hint info >}}
+:::info
+
Paimon will use `hive.metastore.warehouse.dir` in your `hive-site.xml`, please use path with scheme.
For example, `hdfs://...`. Otherwise, Paimon will use the local path.
-{{< /hint >}}
+
+:::
```sql
CREATE CATALOG my_catalog WITH (
@@ -182,9 +197,9 @@ CREATE TABLE word_count (
);
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
**Step 6: Write Data**
diff --git a/docs/content/flink/savepoint.md b/docs/docs/flink/savepoint.md
similarity index 92%
rename from docs/content/flink/savepoint.md
rename to docs/docs/flink/savepoint.md
index 79d0f5daac..e1454149ce 100644
--- a/docs/content/flink/savepoint.md
+++ b/docs/docs/flink/savepoint.md
@@ -1,10 +1,8 @@
---
title: "Savepoint"
-weight: 99
-type: docs
-aliases:
-- /flink/savepoint.html
+sidebar_position: 99
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -61,11 +59,11 @@ to learn how to configure and trigger savepoint.
**Step 3: Choose the tag corresponding to the savepoint.**
The tag corresponding to the savepoint will be named in the form of `savepoint-${savepointID}`. You can refer to
-[Tags Table]({{< ref "concepts/system-tables#tags-table" >}}) to query.
+[Tags Table](../concepts/system-tables#tags-table) to query.
**Step 4: Rollback the paimon table.**
-[Rollback]({{< ref "maintenance/manage-tags#rollback-to-tag" >}}) the Paimon table to the specified tag.
+[Rollback](../maintenance/manage-tags#rollback-to-tag) the Paimon table to the specified tag.
**Step 5: Restart from the savepoint.**
diff --git a/docs/content/flink/sql-alter.md b/docs/docs/flink/sql-alter.md
similarity index 97%
rename from docs/content/flink/sql-alter.md
rename to docs/docs/flink/sql-alter.md
index a21b5b227f..8c72a98eb6 100644
--- a/docs/content/flink/sql-alter.md
+++ b/docs/docs/flink/sql-alter.md
@@ -1,10 +1,8 @@
---
title: "SQL Alter"
-weight: 7
-type: docs
-aliases:
-- /flink/sql-alter.html
+sidebar_position: 7
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -70,17 +68,21 @@ The following SQL rename the table name to new name.
ALTER TABLE my_table RENAME TO my_table_new;
```
-{{< hint info >}}
+:::info
+
If you use object storage without REST Catalog, such as S3 or OSS, please use this syntax carefully, because the renaming of object storage is not atomic, and only partial files may be moved in case of failure.
-{{< /hint >}}
+
+:::
## Adding New Columns
The following SQL adds two columns `c1` and `c2` to table `my_table`.
-{{< hint info >}}
+:::info
+
To add a column in a row type, see [Changing Column Type](#changing-column-type).
-{{< /hint >}}
+
+:::
```sql
ALTER TABLE my_table ADD (c1 INT, c2 STRING);
@@ -102,9 +104,11 @@ The following SQL drops two columns `c1` and `c2` from table `my_table`.
ALTER TABLE my_table DROP (c1, c2);
```
-{{< hint info >}}
+:::info
+
To drop a column in a row type, see [Changing Column Type](#changing-column-type).
-{{< /hint >}}
+
+:::
In hive catalog, you need to ensure:
@@ -145,9 +149,11 @@ SET 'table.exec.sink.not-null-enforcer' = 'DROP';
ALTER TABLE my_table MODIFY coupon_info FLOAT NOT NULL;
```
-{{< hint info >}}
+:::info
+
Changing nullable column to NOT NULL is only supported by Flink currently.
-{{< /hint >}}
+
+:::
## Changing Column Comment
diff --git a/docs/content/flink/sql-ddl.md b/docs/docs/flink/sql-ddl.md
similarity index 94%
rename from docs/content/flink/sql-ddl.md
rename to docs/docs/flink/sql-ddl.md
index 21845a59be..7708af840a 100644
--- a/docs/content/flink/sql-ddl.md
+++ b/docs/docs/flink/sql-ddl.md
@@ -1,10 +1,8 @@
---
title: "SQL DDL"
-weight: 2
-type: docs
-aliases:
-- /flink/sql-ddl.html
+sidebar_position: 2
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -34,7 +32,7 @@ Paimon catalogs currently support three types of metastores:
* `hive` metastore, which additionally stores metadata in Hive metastore. Users can directly access the tables from Hive.
* `jdbc` metastore, which additionally stores metadata in relational databases such as MySQL, Postgres, etc.
-See [CatalogOptions]({{< ref "maintenance/configurations#catalogoptions" >}}) for detailed options when creating a catalog.
+See [CatalogOptions](../maintenance/configurations#catalogoptions) for detailed options when creating a catalog.
### Create Filesystem Catalog
@@ -85,7 +83,7 @@ USE CATALOG my_hive;
You can define any default table options with the prefix `table-default.` for tables created in the catalog.
-Also, you can create [FlinkGenericCatalog]({{< ref "flink/quick-start" >}}).
+Also, you can create [FlinkGenericCatalog](./quick-start).
> When using hive catalog to change incompatible column types through alter table, you need to configure `hive.metastore.disallow.incompatible.col.type.changes=false`. see [HIVE-17832](https://issues.apache.org/jira/browse/HIVE-17832).
@@ -101,7 +99,7 @@ Also, you can create [FlinkGenericCatalog]({{< ref "flink/quick-start" >}}).
By default, Paimon does not synchronize newly created partitions into Hive metastore. Users will see an unpartitioned table in Hive. Partition push-down will be carried out by filter push-down instead.
-If you want to see a partitioned table in Hive and also synchronize newly created partitions into Hive metastore, please set the table property `metastore.partitioned-table` to true. Also see [CoreOptions]({{< ref "maintenance/configurations#coreoptions" >}}).
+If you want to see a partitioned table in Hive and also synchronize newly created partitions into Hive metastore, please set the table property `metastore.partitioned-table` to true. Also see [CoreOptions](../maintenance/configurations#coreoptions).
#### Adding Parameters to a Hive Table
@@ -114,7 +112,7 @@ For instance, using the option `hive.table.owner=Jon` will automatically add the
If you are using an object storage , and you don't want that the location of paimon table/database is accessed by the filesystem of hive,
which may lead to the error such as "No FileSystem for scheme: s3a".
You can set location in the properties of table/database by the config of `location-in-properties`. See
-[setting the location of table/database in properties ]({{< ref "maintenance/configurations#hivecatalogoptions" >}})
+[setting the location of table/database in properties ](../maintenance/configurations#hivecatalogoptions)
### Creating JDBC Catalog
@@ -182,13 +180,17 @@ CREATE TABLE my_table (
) PARTITIONED BY (dt, hh);
```
-{{< hint info >}}
-If you need cross partition upsert (primary keys not contain all partition fields), see [Cross partition Upsert]({{< ref "primary-key-table/data-distribution#cross-partitions-upsert-dynamic-bucket-mode">}}) mode.
-{{< /hint >}}
+:::info
+
+If you need cross partition upsert (primary keys not contain all partition fields), see [Cross partition Upsert](../primary-key-table/data-distribution#cross-partitions-upsert) mode.
+
+:::
+
+:::info
+
+By configuring [partition.expiration-time](../maintenance/manage-partitions), expired partitions can be automatically deleted.
-{{< hint info >}}
-By configuring [partition.expiration-time]({{< ref "maintenance/manage-partitions" >}}), expired partitions can be automatically deleted.
-{{< /hint >}}
+:::
### Specify Statistics Mode
diff --git a/docs/content/flink/sql-lookup.md b/docs/docs/flink/sql-lookup.mdx
similarity index 95%
rename from docs/content/flink/sql-lookup.md
rename to docs/docs/flink/sql-lookup.mdx
index 2e2e0e30ed..215bb103f1 100644
--- a/docs/content/flink/sql-lookup.md
+++ b/docs/docs/flink/sql-lookup.mdx
@@ -1,10 +1,11 @@
---
title: "SQL Lookup"
-weight: 6
-type: docs
-aliases:
-- /flink/sql-lookup.html
+sidebar_position: 6
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -114,11 +115,13 @@ FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;
```
-{{< hint info >}}
+:::info
+
If the main table (`orders`) is CDC stream, `allow_unordered` will be ignored by Flink SQL (only supports append stream),
your streaming job may be blocked. You can try to use `audit_log` system table feature of Paimon to walk around
(convert CDC stream to append stream).
-{{< /hint >}}
+
+:::
## Large Scale Lookup (Fixed Bucket)
@@ -184,21 +187,21 @@ makes no sense.
You can run a Flink Streaming Job to start query service for the table. When QueryService exists, Flink Lookup Join
will prioritize obtaining data from it, which will effectively improve query performance.
-{{< tabs "query-service" >}}
+<Tabs groupId="query-service">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
```sql
CALL sys.query_service('database_name.table_name', parallelism);
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action" >}}
+<TabItem value="flink-action" label="Flink Action">
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
query_service \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -207,6 +210,6 @@ CALL sys.query_service('database_name.table_name', parallelism);
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
diff --git a/docs/content/flink/sql-query.md b/docs/docs/flink/sql-query.mdx
similarity index 90%
rename from docs/content/flink/sql-query.md
rename to docs/docs/flink/sql-query.mdx
index c974527358..ef2e3e9ae6 100644
--- a/docs/content/flink/sql-query.md
+++ b/docs/docs/flink/sql-query.mdx
@@ -1,10 +1,12 @@
---
title: "SQL Query"
-weight: 4
-type: docs
-aliases:
-- /flink/sql-query.html
+sidebar_position: 4
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Label from '@site/src/components/Label';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -41,9 +43,10 @@ SET 'execution.runtime-mode' = 'batch';
Paimon batch reads with time travel can specify a snapshot or a tag and read the corresponding data.
-{{< tabs "time-travel-example" >}}
+<Tabs groupId="time-travel-example">
+
+<TabItem value="flink-dynamic-option" label="Flink (dynamic option)">
-{{< tab "Flink (dynamic option)" >}}
```sql
-- read the snapshot with id 1L
SELECT * FROM t /*+ OPTIONS('scan.snapshot-id' = '1') */;
@@ -61,9 +64,10 @@ SELECT * FROM t /*+ OPTIONS('scan.tag-name' = 'my-tag') */;
-- read the snapshot from watermark, will match the first snapshot after the watermark
SELECT * FROM t /*+ OPTIONS('scan.watermark' = '1678883047356') */;
```
-{{< /tab >}}
-{{< tab "Flink 1.18+" >}}
+</TabItem>
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
Flink SQL supports time travel syntax after 1.18.
@@ -75,9 +79,9 @@ SELECT * FROM t FOR SYSTEM_TIME AS OF TIMESTAMP '2023-01-01 00:00:00';
SELECT * FROM t FOR SYSTEM_TIME AS OF TIMESTAMP '2023-01-01 00:00:00' + INTERVAL '1' DAY
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
### Batch Incremental
@@ -130,9 +134,11 @@ and continue to read the latest changes.
Paimon by default ensures that your startup is properly processed with all data included.
-{{< hint warning >}}
+:::warning
+
Paimon Source in Streaming mode is unbounded, like a queue that never ends.
-{{< /hint >}}
+
+:::
```sql
-- Flink SQL
@@ -155,8 +161,10 @@ SELECT * FROM t WHERE dt > '2023-06-26';
If it's not a partitioned table, or you can't filter by partition, you can use Time travel's stream read.
-{{< tabs "streaming-time-travel" >}}
-{{< tab "Flink (dynamic option)" >}}
+<Tabs groupId="streaming-time-travel">
+
+<TabItem value="flink-dynamic-option" label="Flink (dynamic option)">
+
```sql
-- read changes from snapshot id 1L
SELECT * FROM t /*+ OPTIONS('scan.snapshot-id' = '1') */;
@@ -167,9 +175,11 @@ SELECT * FROM t /*+ OPTIONS('scan.timestamp-millis' = '1678883047356') */;
-- read snapshot id 1L upon first startup, and continue to read the changes
SELECT * FROM t /*+ OPTIONS('scan.mode'='from-snapshot-full','scan.snapshot-id' = '1') */;
```
-{{< /tab >}}
-{{< tab "Flink 1.18+" >}}
+</TabItem>
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
+
Flink SQL supports time travel syntax after 1.18.
```sql
-- read the snapshot from specified timestamp
@@ -178,9 +188,10 @@ SELECT * FROM t FOR SYSTEM_TIME AS OF TIMESTAMP '2023-01-01 00:00:00';
-- you can also use some simple expressions (see flink document to get supported functions)
SELECT * FROM t FOR SYSTEM_TIME AS OF TIMESTAMP '2023-01-01 00:00:00' + INTERVAL '1' DAY
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
Time travel's stream read rely on snapshots, but by default, snapshot only retains data within 1 hour, which can
prevent you from reading older incremental data. So, Paimon also provides another mode for streaming reads,
@@ -204,31 +215,31 @@ Disable `scan.infer-parallelism`, global parallelism will be used for reads.
You can also manually specify the parallelism from `scan.parallelism`.
-<table class="configuration table table-bordered">
+<table className="configuration table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Key</th>
+ <th className="text-left" style={{width: "15%"}}>Default</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "55%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>scan.infer-parallelism</h5></td>
- <td style="word-wrap: break-word;">true</td>
+ <td style={{wordWrap: "break-word"}}>true</td>
<td>Boolean</td>
<td>If it is false, parallelism of source are set by global parallelism. Otherwise, source parallelism is inferred from splits number (batch mode) or bucket number(streaming mode).</td>
</tr>
<tr>
<td><h5>scan.infer-parallelism.max</h5></td>
- <td style="word-wrap: break-word;">1024</td>
+ <td style={{wordWrap: "break-word"}}>1024</td>
<td>Integer</td>
<td>If scan.infer-parallelism is true, limit the parallelism of source through this option.</td>
</tr>
<tr>
<td><h5>scan.parallelism</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>Integer</td>
<td>Define a custom parallelism for the scan source. By default, if this option is not defined, the planner will derive the parallelism for each statement individually by also considering the global configuration. If user enable the scan.infer-parallelism, the planner will derive the parallelism by inferred parallelism.</td>
</tr>
@@ -237,7 +248,7 @@ You can also manually specify the parallelism from `scan.parallelism`.
## Query Optimization
-{{< label Batch >}}{{< label Streaming >}}
+<Label>Batch</Label><Label>Streaming</Label>
It is highly recommended to specify partition and primary key filters
along with the query, which will speed up the data skipping of the query.
diff --git a/docs/content/flink/sql-write.md b/docs/docs/flink/sql-write.mdx
similarity index 86%
rename from docs/content/flink/sql-write.md
rename to docs/docs/flink/sql-write.mdx
index cf468e7699..c34211f8de 100644
--- a/docs/content/flink/sql-write.md
+++ b/docs/docs/flink/sql-write.mdx
@@ -1,10 +1,11 @@
---
title: "SQL Write"
-weight: 2
-type: docs
-aliases:
-- /flink/sql-write.html
+sidebar_position: 2
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -47,13 +48,13 @@ INSERT INTO my_table SELECT ...
`INSERT INTO` supports both batch and streaming mode. In Streaming mode, by default, it will also perform compaction,
snapshot expiration, and even partition expiration in Flink Sink (if it is configured).
-For multiple jobs to write the same table, you can refer to [dedicated compaction job]({{< ref "maintenance/dedicated-compaction#dedicated-compaction-job" >}}) for more info.
+For multiple jobs to write the same table, you can refer to [dedicated compaction job](../maintenance/dedicated-compaction#dedicated-compaction-job) for more info.
### Clustering
-In Paimon, clustering is a feature that allows you to cluster data in your [Append Table]({{< ref "append-table/overview" >}})
+In Paimon, clustering is a feature that allows you to cluster data in your [Append Table](../append-table/overview)
based on the values of certain columns during the write process. This organization of data can significantly enhance the efficiency of downstream
-tasks when reading the data, as it enables faster and more targeted data retrieval. This feature is only supported for [Append Table]({{< ref "append-table/overview" >}})(bucket = -1)
+tasks when reading the data, as it enables faster and more targeted data retrieval. This feature is only supported for [Append Table](../append-table/overview)(bucket = -1)
and batch execution mode.
To utilize clustering, you can specify the columns you want to cluster when creating or writing to a table. Here's a simple example of how to enable clustering:
@@ -79,7 +80,7 @@ The data is clustered using an automatically chosen strategy (such as ORDER, ZOR
by setting the `sink.clustering.strategy`. Clustering relies on sampling and sorting. If the clustering process takes too much time, you can decrease
the total sample number by setting the `sink.clustering.sample-factor` or disable the sorting step by setting the `sink.clustering.sort-in-cluster` to false.
-You can refer to [FlinkConnectorOptions]({{< ref "maintenance/configurations#flinkconnectoroptions" >}}) for more info about the configurations above.
+You can refer to [FlinkConnectorOptions](../maintenance/configurations#flinkconnectoroptions) for more info about the configurations above.
## Overwriting the Whole Table
@@ -119,9 +120,9 @@ INSERT OVERWRITE my_table /*+ OPTIONS('dynamic-partition-overwrite' = 'false') *
## Truncate tables
-{{< tabs "truncate-tables-syntax" >}}
+<Tabs groupId="truncate-tables-syntax">
-{{< tab "Flink 1.17-" >}}
+<TabItem value="flink-1-17" label="Flink 1.17-">
You can use `INSERT OVERWRITE` to purge tables by inserting empty value.
@@ -129,17 +130,17 @@ You can use `INSERT OVERWRITE` to purge tables by inserting empty value.
INSERT OVERWRITE my_table /*+ OPTIONS('dynamic-partition-overwrite'='false') */ SELECT * FROM my_table WHERE false;
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink 1.18+" >}}
+<TabItem value="flink-1-18" label="Flink 1.18+">
```sql
TRUNCATE TABLE my_table;
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Purging Partitions
@@ -173,13 +174,15 @@ PARTITION (k0 = 0, k1 = 0) SELECT v FROM my_table WHERE false;
## Updating tables
-{{< hint info >}}
+:::info
+
Important table properties setting:
-1. Only [primary key table]({{< ref "primary-key-table/overview" >}}) supports this feature.
-2. [MergeEngine]({{< ref "primary-key-table/merge-engine" >}}) needs to be [deduplicate]({{< ref "primary-key-table/merge-engine/overview#deduplicate" >}})
- or [partial-update]({{< ref "primary-key-table/merge-engine/partial-update" >}}) to support this feature.
+1. Only [primary key table](../primary-key-table/overview) supports this feature.
+2. [MergeEngine](../primary-key-table/merge-engine/) needs to be [deduplicate](../primary-key-table/merge-engine/overview#deduplicate)
+ or [partial-update](../primary-key-table/merge-engine/partial-update) to support this feature.
3. Do not support updating primary keys.
-{{< /hint >}}
+
+:::
Currently, Paimon supports updating records by using `UPDATE` in Flink 1.17 and later versions. You can perform `UPDATE` in Flink's `batch` mode.
@@ -204,18 +207,20 @@ UPDATE my_table SET b = 1, c = 2 WHERE a = 'myTable';
## Deleting from table
-{{< tabs "delete-from-table" >}}
+<Tabs groupId="delete-from-table">
+
+<TabItem value="flink-1-17" label="Flink 1.17+">
-{{< tab "Flink 1.17+" >}}
+:::info
-{{< hint info >}}
Important table properties setting:
1. Only primary key tables support this feature.
-2. If the table has primary keys, the following [MergeEngine]({{< ref "primary-key-table/merge-engine/overview" >}}) support this feature:
- * [deduplicate]({{< ref "primary-key-table/merge-engine/overview#deduplicate" >}}).
- * [partial-update]({{< ref "primary-key-table/merge-engine/partial-update" >}}) with option 'partial-update.remove-record-on-delete' enabled.
+2. If the table has primary keys, the following [MergeEngine](../primary-key-table/merge-engine/overview) support this feature:
+ * [deduplicate](../primary-key-table/merge-engine/overview#deduplicate).
+ * [partial-update](../primary-key-table/merge-engine/partial-update) with option 'partial-update.remove-record-on-delete' enabled.
3. Do not support deleting from table in streaming mode.
-{{< /hint >}}
+
+:::
```sql
-- Syntax
@@ -237,9 +242,9 @@ CREATE TABLE my_table (
DELETE FROM my_table WHERE currency = 'UNKNOWN';
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Partition Mark Done
diff --git a/docs/content/iceberg/append-table.md b/docs/docs/iceberg/append-table.mdx
similarity index 88%
rename from docs/content/iceberg/append-table.md
rename to docs/docs/iceberg/append-table.mdx
index 495cb508ca..8e5b180138 100644
--- a/docs/content/iceberg/append-table.md
+++ b/docs/docs/iceberg/append-table.mdx
@@ -1,10 +1,11 @@
---
title: "Append Table"
-weight: 2
-type: docs
-aliases:
-- /iceberg/append-table.html
+sidebar_position: 2
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -34,9 +35,10 @@ Please refer to Iceberg's document if you haven't set up Iceberg.
Let's now create a Paimon append only table with Iceberg compatibility enabled and insert some data.
-{{< tabs "create-paimon-append-only-table" >}}
+<Tabs groupId="create-paimon-append-only-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
@@ -52,9 +54,11 @@ CREATE TABLE paimon_catalog.`default`.cities (
INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'new york'), ('germany', 'berlin'), ('usa', 'chicago'), ('germany', 'hamburg');
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
Start `spark-sql` with the following command line.
```bash
@@ -81,15 +85,17 @@ CREATE TABLE paimon_catalog.`default`.cities (
INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'new york'), ('germany', 'berlin'), ('usa', 'chicago'), ('germany', 'hamburg');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
Now let's query this Paimon table with Iceberg connector.
-{{< tabs "query-paimon-append-only-table" >}}
+<Tabs groupId="query-paimon-append-only-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
CREATE CATALOG iceberg_catalog WITH (
'type' = 'iceberg',
@@ -108,9 +114,11 @@ SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
+----+--------------------------------+--------------------------------+
*/
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
```sql
SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
/*
@@ -118,15 +126,17 @@ germany berlin
germany hamburg
*/
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
Let's insert more data and query again.
-{{< tabs "query-paimon-append-only-table-again" >}}
+<Tabs groupId="query-paimon-append-only-table-again">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'houston'), ('germany', 'munich');
@@ -141,9 +151,11 @@ SELECT * FROM iceberg_catalog.`default`.cities WHERE country = 'germany';
+----+--------------------------------+--------------------------------+
*/
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
```sql
INSERT INTO paimon_catalog.`default`.cities VALUES ('usa', 'houston'), ('germany', 'munich');
@@ -154,7 +166,8 @@ germany berlin
germany hamburg
*/
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
diff --git a/docs/content/iceberg/configurations.md b/docs/docs/iceberg/configurations.mdx
similarity index 79%
rename from docs/content/iceberg/configurations.md
rename to docs/docs/iceberg/configurations.mdx
index daeaf5fd69..cd20534b62 100644
--- a/docs/content/iceberg/configurations.md
+++ b/docs/docs/iceberg/configurations.mdx
@@ -1,10 +1,11 @@
---
title: "Configurations"
-weight: 7
-type: docs
-aliases:
-- /iceberg/configurations.html
+sidebar_position: 7
---
+
+import ConfigTable from '@site/src/components/ConfigTable';
+import icebergConfigurationHtml from '@site/generated/iceberg_configuration.html';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -28,4 +29,4 @@ under the License.
Options for Iceberg Compatibility.
-{{< generated/iceberg_configuration >}}
+<ConfigTable html={icebergConfigurationHtml} />
diff --git a/docs/content/iceberg/ecosystem.md b/docs/docs/iceberg/ecosystem.mdx
similarity index 93%
rename from docs/content/iceberg/ecosystem.md
rename to docs/docs/iceberg/ecosystem.mdx
index 846209419e..a111ae8f4f 100644
--- a/docs/content/iceberg/ecosystem.md
+++ b/docs/docs/iceberg/ecosystem.mdx
@@ -1,10 +1,11 @@
---
title: "Ecosystem"
-weight: 6
-type: docs
-aliases:
-- /iceberg/ecosystem.html
+sidebar_position: 6
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -45,9 +46,10 @@ See [Trino's document](https://trino.io/docs/current/connector/iceberg.html#gene
Let's first create a Paimon table with Iceberg compatibility enabled.
-{{< tabs "paimon-append-only-table-trino-1" >}}
+<Tabs groupId="paimon-append-only-table-trino-1">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
@@ -64,9 +66,11 @@ CREATE TABLE paimon_catalog.`default`.animals (
INSERT INTO paimon_catalog.`default`.animals VALUES ('mammal', 'cat'), ('mammal', 'dog'), ('reptile', 'snake'), ('reptile', 'lizard');
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
Start `spark-sql` with the following command line.
```bash
@@ -94,9 +98,10 @@ CREATE TABLE paimon_catalog.`default`.animals (
INSERT INTO paimon_catalog.`default`.animals VALUES ('mammal', 'cat'), ('mammal', 'dog'), ('reptile', 'snake'), ('reptile', 'lizard');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
Start Trino using Iceberg catalog and query from Paimon table.
diff --git a/docs/content/iceberg/hive-catalog.md b/docs/docs/iceberg/hive-catalog.md
similarity index 98%
rename from docs/content/iceberg/hive-catalog.md
rename to docs/docs/iceberg/hive-catalog.md
index c39c981291..52bd5d6f2a 100644
--- a/docs/content/iceberg/hive-catalog.md
+++ b/docs/docs/iceberg/hive-catalog.md
@@ -1,10 +1,8 @@
---
title: "Hive Catalogs"
-weight: 5
-type: docs
-aliases:
-- /iceberg/hive-catalogs.html
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/iceberg/iceberg-tags.md b/docs/docs/iceberg/iceberg-tags.md
similarity index 96%
rename from docs/content/iceberg/iceberg-tags.md
rename to docs/docs/iceberg/iceberg-tags.md
index 26071add3d..fd8159d65f 100644
--- a/docs/content/iceberg/iceberg-tags.md
+++ b/docs/docs/iceberg/iceberg-tags.md
@@ -1,10 +1,8 @@
---
title: "Iceberg Tags"
-weight: 4
-type: docs
-aliases:
-- /iceberg/iceberg-tags.html
+sidebar_position: 4
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/iceberg/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/iceberg/index.md
index eacaa231ca..34a3b1bb1b 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/iceberg/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Iceberg Metadata"
+sidebar_position: 98
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/iceberg/overview.md b/docs/docs/iceberg/overview.md
similarity index 98%
rename from docs/content/iceberg/overview.md
rename to docs/docs/iceberg/overview.md
index 35dc351905..2519d32a0a 100644
--- a/docs/content/iceberg/overview.md
+++ b/docs/docs/iceberg/overview.md
@@ -1,10 +1,8 @@
---
title: "Overview"
-weight: 1
-type: docs
-aliases:
-- /iceberg/overview.html
+sidebar_position: 1
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -104,8 +102,10 @@ Paimon Iceberg compatibility currently supports the following data types.
| `MAP` | `map` |
| `ROW` | `struct` |
-{{< hint info >}}
+:::info
+
**Note on Timestamp Types:**
- `TIMESTAMP` and `TIMESTAMP_LTZ` types with precision from 3 to 6 are mapped to standard Iceberg timestamp types
- `TIMESTAMP` and `TIMESTAMP_LTZ` types with precision from 7 to 9 use nanosecond precision and require Iceberg v3 format
-{{< /hint >}}
+
+:::
diff --git a/docs/content/iceberg/primary-key-table.md b/docs/docs/iceberg/primary-key-table.mdx
similarity index 91%
rename from docs/content/iceberg/primary-key-table.md
rename to docs/docs/iceberg/primary-key-table.mdx
index 1c34b6d2de..fb9a23223b 100644
--- a/docs/content/iceberg/primary-key-table.md
+++ b/docs/docs/iceberg/primary-key-table.mdx
@@ -1,10 +1,11 @@
---
title: "Primary Key Table"
-weight: 3
-type: docs
-aliases:
-- /iceberg/primary-key-table.html
+sidebar_position: 3
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -32,9 +33,10 @@ Please refer to Iceberg's document if you haven't set up Iceberg.
* Flink: [Preparation when using Flink SQL Client](https://iceberg.apache.org/docs/latest/flink/#preparation-when-using-flink-sql-client)
* Spark: [Using Iceberg in Spark 3](https://iceberg.apache.org/docs/latest/spark-getting-started/#using-iceberg-in-spark-3)
-{{< tabs "paimon-primary-key-table" >}}
+<Tabs groupId="paimon-primary-key-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
@@ -81,9 +83,11 @@ SELECT * FROM iceberg_catalog.`default`.orders WHERE status = 'COMPLETED';
+----+----------------------+--------------------------------+--------------------------------+
*/
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
Start `spark-sql` with the following command line.
```bash
@@ -126,9 +130,10 @@ SELECT * FROM iceberg_catalog.`default`.orders WHERE status = 'COMPLETED';
1 COMPLETED 100.0
*/
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
Paimon primary key tables organize data files as LSM trees, so data files must be merged in memory before querying.
However, Iceberg readers are not able to merge data files, so they can only query data files on the highest level of LSM trees.
@@ -138,25 +143,25 @@ So **to conclude, for primary key tables, Iceberg readers can only query data af
By default, there is no guarantee on how frequently Paimon will perform full compaction.
You can configure the following table option, so that Paimon is forced to perform full compaction after several commits.
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Option</th>
- <th class="text-left" style="width: 5%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 60%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Option</th>
+ <th className="text-left" style={{width: "5%"}}>Default</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "60%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>compaction.optimization-interval</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>Duration</td>
<td>Full compaction will be constantly triggered per time interval. First compaction after the job starts will always be full compaction.</td>
</tr>
<tr>
<td><h5>full-compaction.delta-commits</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
+ <td style={{wordWrap: "break-word"}}>(none)</td>
<td>Integer</td>
<td>Full compaction will be constantly triggered after delta commits. Only implemented in Flink.</td>
</tr>
@@ -168,7 +173,7 @@ We recommend full compaction to be performed once or twice per hour.
## Deletion Vector Support
-[Deletion vectors]({{< ref "concepts/spec/tableindex#deletion-vectors" >}}) in Paimon are used to store deleted records for each file.
+[Deletion vectors](../concepts/spec/tableindex#deletion-vectors) in Paimon are used to store deleted records for each file.
Under deletion-vector mode, paimon readers can directly filter out unnecessary records during reading phase without merging data.
Fortunately, Iceberg has supported [deletion vectors](https://iceberg.apache.org/spec/?h=deletion#deletion-vectors) in [Version 3](https://iceberg.apache.org/spec/?h=deletion#version-3).
This means that if the Iceberg reader can recognize Paimon's deletion vectors, it will be able to read all of Paimon's data, even without the ability to merge data files.
@@ -182,9 +187,10 @@ If the following conditions are met, it will construct metadata about Paimon's d
* JDK version should be 11+. Iceberg has stopped supporting JDK 8 since version 1.7.0.
Here is an example:
-{{< tabs "deletion-vector-table" >}}
+<Tabs groupId="deletion-vector-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
-- flink version: 1.20.1
@@ -262,28 +268,29 @@ SELECT * FROM iceberg_catalog.`default`.T;
*/
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
-{{< hint info >}}
+:::info
note1: Upgrade the implementation of deletion vector to 64-bit bitmap if necessary.
-{{< /hint >}}
+:::
If your paimon table has already been in deletion-vector mode, but 32-bit bitmap was used for deletion vector.
You need to upgrade the implementation of deletion vector to 64-bit bitmap if you want to synchronize deletion-vector metadata to iceberg.
You can follow the following steps to upgrade to 64-bit deletion-vector:
1. stop all the writing jobs of your paimon table.
-2. perform a [full compaction]({{< ref "maintenance/dedicated-compaction#dedicated-compaction-job" >}}) to your paimon table.
+2. perform a [full compaction](../maintenance/dedicated-compaction#dedicated-compaction-job) to your paimon table.
3. run `ALTER TABLE tableName SET ('deletion-vectors.bitmap64' = 'true')` to upgrade to 64-bit deletion vector.
4. restart your writing job. If meeting the all the conditions mentioned above, deletion vector metadata will be synchronized to iceberg.
-{{< hint info >}}
+:::info
note2: Upgrade the format version of iceberg to 3 if necessary.
-{{< /hint >}}
+:::
You can upgrade the format version of iceberg from 2 to 3 by setting `'metadata.iceberg.format-version' = '3'`.
This will recreate the iceberg metadata without using the base metadata.
diff --git a/docs/content/iceberg/rest-catalog.md b/docs/docs/iceberg/rest-catalog.mdx
similarity index 88%
rename from docs/content/iceberg/rest-catalog.md
rename to docs/docs/iceberg/rest-catalog.mdx
index 8acdd216ba..547d979263 100644
--- a/docs/content/iceberg/rest-catalog.md
+++ b/docs/docs/iceberg/rest-catalog.mdx
@@ -1,10 +1,11 @@
---
title: "Rest Catalog"
-weight: 5
-type: docs
-aliases:
-- /iceberg/rest-catalog.html
+sidebar_position: 5
---
+
+import Stable from '@site/src/components/Stable';
+import Unstable from '@site/src/components/Unstable';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -38,28 +39,36 @@ and use this rest catalog to commit metadata.
This feature needs dependency:
-{{< stable >}}
+<Stable>
+
+
+
+[paimon-iceberg-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-iceberg/@@VERSION@@/paimon-iceberg-@@VERSION@@.jar),
+
+
+
+</Stable>
+
+<Unstable>
+
-[paimon-iceberg-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-iceberg/{{< version >}}/paimon-iceberg-{{< version >}}.jar),
-{{< /stable >}}
+[paimon-iceberg-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-iceberg/@@VERSION@@/),
-{{< unstable >}}
-[paimon-iceberg-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-iceberg/{{< version >}}/),
-{{< /unstable >}}
+</Unstable>
and JDK version should be 11+.
You can also manually build the jar from the source code.(need JDK 11+)
-To build from source code, [clone the git repository]({{< github_repo >}}).
+To build from source code, [clone the git repository](@@GITHUB_REPO@@).
Build bundled jar with the following command.
- `mvn clean install -DskipTests`
-You can find the jar in `./paimon-iceberg/target/paimon-iceberg-{{< version >}}.jar`.
+You can find the jar in `./paimon-iceberg/target/paimon-iceberg-@@VERSION@@.jar`.
**Example:**
diff --git a/docs/docs/index.mdx b/docs/docs/index.mdx
new file mode 100644
index 0000000000..72e26dda4b
--- /dev/null
+++ b/docs/docs/index.mdx
@@ -0,0 +1,133 @@
+---
+title: "Apache Paimon"
+hide_table_of_contents: true
+slug: /
+displayed_sidebar: docs
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<div className="hero-section">
+<h1>Apache Paimon</h1>
+<p className="hero-subtitle">
+Data Lake Platform — unified batch, streaming, and multimodal AI in a single lake format.
+</p>
+</div>
+
+<div className="card-grid">
+
+<a className="nav-card" href="concepts/overview">
+ <div className="nav-card-icon">📖</div>
+ <div className="nav-card-body">
+ <h3>Concepts</h3>
+ <p>Core architecture, data types, catalogs, and system tables</p>
+ </div>
+</a>
+
+<a className="nav-card" href="primary-key-table/overview">
+ <div className="nav-card-icon">🔑</div>
+ <div className="nav-card-body">
+ <h3>Table with PK</h3>
+ <p>Merge engines, changelog producers, compaction, and streaming updates</p>
+ </div>
+</a>
+
+<a className="nav-card" href="append-table/overview">
+ <div className="nav-card-icon">📋</div>
+ <div className="nav-card-body">
+ <h3>Table w/o PK</h3>
+ <p>Append-only tables, incremental clustering, blob & vector support</p>
+ </div>
+</a>
+
+<a className="nav-card" href="flink/quick-start">
+ <div className="nav-card-icon">⚡</div>
+ <div className="nav-card-body">
+ <h3>Flink</h3>
+ <p>Quick start, SQL DDL/DML, streaming read & write, procedures</p>
+ </div>
+</a>
+
+<a className="nav-card" href="spark/quick-start">
+ <div className="nav-card-icon">✨</div>
+ <div className="nav-card-body">
+ <h3>Spark</h3>
+ <p>Quick start, SQL operations, DataFrames, structured streaming</p>
+ </div>
+</a>
+
+<a className="nav-card" href="pypaimon/overview">
+ <div className="nav-card-icon">🐍</div>
+ <div className="nav-card-body">
+ <h3>PyPaimon & AI</h3>
+ <p>Python SDK, Ray, PyTorch, Pandas integration for AI & multimodal workloads</p>
+ </div>
+</a>
+
+<a className="nav-card" href="ecosystem/overview">
+ <div className="nav-card-icon">🔗</div>
+ <div className="nav-card-body">
+ <h3>Ecosystem</h3>
+ <p>StarRocks, Doris, Hive, Trino, Presto integrations</p>
+ </div>
+</a>
+
+<a className="nav-card" href="cdc-ingestion/overview">
+ <div className="nav-card-icon">🔄</div>
+ <div className="nav-card-body">
+ <h3>CDC Ingestion</h3>
+ <p>MySQL, PostgreSQL, Kafka, MongoDB, Pulsar CDC pipelines</p>
+ </div>
+</a>
+
+<a className="nav-card" href="maintenance/configurations">
+ <div className="nav-card-icon">⚙️</div>
+ <div className="nav-card-body">
+ <h3>Maintenance</h3>
+ <p>Configuration, snapshots, tags, metrics, and performance tuning</p>
+ </div>
+</a>
+
+</div>
+
+---
+
+## Why Paimon?
+
+<div className="feature-grid">
+<div className="feature-item">
+
+### Large-Scale Analytics
+Petabyte-scale tables with time travel, fast scan planning, schema evolution, and incremental clustering.
+
+</div>
+<div className="feature-item">
+
+### Realtime Streaming
+LSM-powered streaming updates with multiple merge engines (Deduplicate, Partial Update, Aggregation) and changelog producers.
+
+</div>
+<div className="feature-item">
+
+### Multimodal AI
+Store images, videos, embeddings in blob tables. Vector search (DiskANN), full-text search, and native Python SDK for ML pipelines.
+
+</div>
+</div>
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/learn-paimon/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/learn-paimon/index.md
index eacaa231ca..68ae077070 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/learn-paimon/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Learn Paimon"
+sidebar_position: 100
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/learn-paimon/scenario-guide.md b/docs/docs/learn-paimon/scenario-guide.mdx
similarity index 90%
rename from docs/content/learn-paimon/scenario-guide.md
rename to docs/docs/learn-paimon/scenario-guide.mdx
index a5b64e2bca..57866da5d1 100644
--- a/docs/content/learn-paimon/scenario-guide.md
+++ b/docs/docs/learn-paimon/scenario-guide.mdx
@@ -1,10 +1,11 @@
---
title: "Scenario Guide"
-weight: 2
-type: docs
-aliases:
-- /learn-paimon/scenario-guide.html
+sidebar_position: 2
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -52,7 +53,7 @@ configurations that are suited for different scenarios.
## Primary Key Table
Use a Primary Key Table when your data has a natural unique key and you need **real-time updates** (insert, update, delete).
-See [Primary Key Table Overview]({{< ref "primary-key-table/overview" >}}).
+See [Primary Key Table Overview](../primary-key-table/overview).
### Scenario 1: CDC Real-Time Sync
@@ -79,10 +80,10 @@ CREATE TABLE orders (
**Why this configuration:**
-- **`deletion-vectors.enabled = true`** (MOW mode): Enables [Merge On Write]({{< ref "primary-key-table/table-mode#merge-on-write" >}}) with Deletion Vectors.
+- **`deletion-vectors.enabled = true`** (MOW mode): Enables [Merge On Write](../primary-key-table/table-mode#merge-on-write) with Deletion Vectors.
This mode gives you the best balance of write and read performance. Compared to the default MOR mode, MOW
avoids merging at read time, which greatly improves OLAP query performance.
-- **`changelog-producer = lookup`**: Generates a complete [changelog]({{< ref "primary-key-table/changelog-producer#lookup" >}})
+- **`changelog-producer = lookup`**: Generates a complete [changelog](../primary-key-table/changelog-producer#lookup)
for downstream streaming consumers. If your CDC source is directly connected to a database (e.g., MySQL CDC, Postgres CDC),
you can use `changelog-producer = input` instead, since the database CDC stream already provides a complete changelog.
However, if your CDC source comes from Kafka (or other message queues), `input` may not be reliable — use `lookup` to
@@ -92,7 +93,7 @@ CREATE TABLE orders (
on data volume. If you are sensitive to data visibility latency, set a fixed bucket number (e.g. `'bucket' = '5'`)
— roughly 1 bucket per 1GB of data in a partition.
-**CDC Ingestion Tip:** Use [Paimon CDC Ingestion]({{< ref "cdc-ingestion/overview" >}}) for whole-database sync with
+**CDC Ingestion Tip:** Use [Paimon CDC Ingestion](../cdc-ingestion/overview) for whole-database sync with
automatic table creation and schema evolution support.
### Scenario 2: Multi-Stream Partial Column Updates
@@ -122,7 +123,7 @@ CREATE TABLE order_wide (
);
```
-**Why:** The [partial-update]({{< ref "primary-key-table/merge-engine/partial-update" >}}) merge engine allows each
+**Why:** The [partial-update](../primary-key-table/merge-engine/partial-update) merge engine allows each
stream to update only its own columns without overwriting the others. `sequence-group` ensures ordering within each
stream independently.
@@ -151,7 +152,7 @@ CREATE TABLE product_metrics (
);
```
-**Why:** The [aggregation]({{< ref "primary-key-table/merge-engine/aggregation" >}}) merge engine supports 20+
+**Why:** The [aggregation](../primary-key-table/merge-engine/aggregation) merge engine supports 20+
aggregate functions (`sum`, `max`, `min`, `count`, `hll_sketch`, `theta_sketch`, `collect`, `merge_map`, etc.), ideal
for real-time metric accumulation.
@@ -176,7 +177,7 @@ CREATE TABLE first_login (
);
```
-**Why:** The [first-row]({{< ref "primary-key-table/merge-engine/first-row" >}}) merge engine keeps only the earliest
+**Why:** The [first-row](../primary-key-table/merge-engine/first-row) merge engine keeps only the earliest
record for each primary key and produces insert-only changelog, making it perfect for streaming log deduplication.
### Primary Key Table: Bucket Mode Comparison
@@ -203,7 +204,7 @@ record for each primary key and produces insert-only changelog, making it perfec
Use an Append Table when your data **has no natural primary key**, or you are working with **batch ETL** pipelines
where data is only inserted and does not need upsert semantics.
-See [Append Table Overview]({{< ref "append-table/overview" >}}).
+See [Append Table Overview](../append-table/overview).
Compared to Primary Key Tables, Append Tables have much better batch read/write performance, simpler design, and lower
resource consumption. **We recommend using Append Tables for most batch processing scenarios.**
@@ -236,7 +237,7 @@ Paimon automatically handles small file merging and supports:
- **Incremental Clustering** for advanced data layout optimization.
**Query Optimization Tip:** If your queries frequently filter on specific columns, consider using
-[Incremental Clustering]({{< ref "append-table/incremental-clustering" >}}) to sort data by those columns:
+[Incremental Clustering](../append-table/incremental-clustering) to sort data by those columns:
```sql
ALTER TABLE dwd_events SET (
@@ -284,7 +285,7 @@ SELECT * FROM product_logs WHERE product_id = 12345;
SELECT * FROM product_logs WHERE product_id IN (1, 2, 3);
```
-See [Bucketed Append — Data Skipping]({{< ref "append-table/bucketed#data-skipping" >}}).
+See [Bucketed Append — Data Skipping](../append-table/bucketed#data-skipping).
**Bucketed Join Bonus:** If two bucketed tables share the same `bucket-key` and bucket count, Spark can join them
**without shuffle**, significantly accelerating batch join queries:
@@ -297,7 +298,7 @@ SELECT * FROM product_logs JOIN product_dim
ON product_logs.product_id = product_dim.product_id;
```
-See [Bucketed Join]({{< ref "append-table/bucketed#bucketed-join" >}}).
+See [Bucketed Join](../append-table/bucketed#bucketed-join).
### Scenario 7: Queue-Like Ordered Streaming (Bucketed Append)
@@ -322,7 +323,7 @@ CREATE TABLE event_stream (
**Why:** Within the same bucket, records are strictly ordered by write time. Streaming reads deliver records in exact
write order per bucket. This gives you Kafka-like partitioned ordering at data lake cost.
-See [Bucketed Streaming]({{< ref "append-table/bucketed#bucketed-streaming" >}}).
+See [Bucketed Streaming](../append-table/bucketed#bucketed-streaming).
### Append Table: Bucket Mode Comparison
@@ -337,7 +338,7 @@ See [Bucketed Streaming]({{< ref "append-table/bucketed#bucketed-streaming" >}})
Paimon is a multimodal lakehouse for AI. You can keep multimodal data, metadata, and embeddings in the same table and
query them via vector search, full-text search, or SQL. All multimodal features are built on top of Append Tables with
-[Data Evolution]({{< ref "append-table/data-evolution" >}}) mode enabled.
+[Data Evolution](../append-table/data-evolution) mode enabled.
### Scenario 8: Storing Multimodal Data (Blob Table)
@@ -346,8 +347,10 @@ the data lake, and want efficient column projection without loading large binary
**Recommended Configuration:**
-{{< tabs "blob-scenario" >}}
-{{< tab "Flink SQL" >}}
+<Tabs groupId="blob-scenario">
+
+<TabItem value="flink-sql" label="Flink SQL">
+
```sql
CREATE TABLE image_table (
id INT,
@@ -360,8 +363,11 @@ CREATE TABLE image_table (
'blob-field' = 'image'
);
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
+
```sql
CREATE TABLE image_table (
id INT,
@@ -374,10 +380,12 @@ CREATE TABLE image_table (
'blob-field' = 'image'
);
```
-{{< /tab >}}
-{{< /tabs >}}
-**Why:** The [Blob Storage]({{< ref "append-table/blob" >}}) separates large binary data into dedicated `.blob` files
+</TabItem>
+
+</Tabs>
+
+**Why:** The [Blob Storage](../append-table/blob) separates large binary data into dedicated `.blob` files
while metadata stays in standard columnar files (Parquet/ORC). This means:
- `SELECT id, name, label FROM image_table` does **not** load any blob data — very fast.
@@ -409,8 +417,10 @@ application that needs approximate nearest neighbor (ANN) search on embeddings.
**Recommended Configuration:**
-{{< tabs "vector-scenario" >}}
-{{< tab "Spark SQL" >}}
+<Tabs groupId="vector-scenario">
+
+<TabItem value="spark-sql" label="Spark SQL">
+
```sql
CREATE TABLE doc_embeddings (
doc_id INT,
@@ -426,8 +436,11 @@ CREATE TABLE doc_embeddings (
'vector.file.format' = 'lance'
);
```
-{{< /tab >}}
-{{< tab "Java API" >}}
+
+</TabItem>
+
+<TabItem value="java-api" label="Java API">
+
```java
Schema schema = Schema.newBuilder()
.column("doc_id", DataTypes.INT())
@@ -441,8 +454,10 @@ Schema schema = Schema.newBuilder()
.option("vector.file.format", "lance")
.build();
```
-{{< /tab >}}
-{{< /tabs >}}
+
+</TabItem>
+
+</Tabs>
**Build the vector index and search:**
@@ -461,7 +476,7 @@ SELECT * FROM vector_search('doc_embeddings', 'embedding', array(0.1f, 0.2f, ...
The legacy index type `lumina-vector-ann` is still accepted for existing tables and SQL compatibility.
-**Why:** The [Global Index]({{< ref "append-table/global-index" >}}) with DiskANN provides high-performance ANN search.
+**Why:** The [Global Index](../append-table/global-index) with DiskANN provides high-performance ANN search.
Vector data is stored in dedicated `.vector.lance` files optimized for dense vectors, while scalar columns stay in
Parquet. You can also build a **BTree Index** on scalar columns for efficient filtering:
@@ -509,7 +524,7 @@ ON t.user_id = s.user_id
WHEN MATCHED THEN UPDATE SET t.embedding = s.embedding;
```
-**Why:** [Data Evolution]({{< ref "append-table/data-evolution" >}}) mode writes only the updated columns to new files
+**Why:** [Data Evolution](../append-table/data-evolution) mode writes only the updated columns to new files
and merges them at read time. This is ideal for:
- Adding new feature columns and backfilling data without rewriting the entire table.
@@ -556,14 +571,14 @@ df = table_read.to_pandas(splits)
arrow_table = table_read.to_arrow(splits)
```
-**Why:** [PyPaimon]({{< ref "pypaimon/overview" >}}) is a pure Python SDK (no JDK required) that integrates seamlessly
+**Why:** [PyPaimon](../pypaimon/overview) is a pure Python SDK (no JDK required) that integrates seamlessly
with the Python AI ecosystem:
- **PyTorch**: Direct `DataLoader` integration with streaming and prefetch support.
- **Ray**: Distributed data processing with configurable parallelism.
- **Pandas / PyArrow**: Native DataFrame and Arrow Table support for data science workflows.
- **Data Evolution**: Python API supports `update_by_arrow_with_row_id` and `upsert_by_arrow_with_key` for
- row-level updates from Python. See [PyPaimon Data Evolution]({{< ref "pypaimon/data-evolution" >}}).
+ row-level updates from Python. See [PyPaimon Data Evolution](../pypaimon/data-evolution).
---
diff --git a/docs/content/learn-paimon/understand-files.md b/docs/docs/learn-paimon/understand-files.mdx
similarity index 91%
rename from docs/content/learn-paimon/understand-files.md
rename to docs/docs/learn-paimon/understand-files.mdx
index 58fed51b84..e63b2e647f 100644
--- a/docs/content/learn-paimon/understand-files.md
+++ b/docs/docs/learn-paimon/understand-files.mdx
@@ -1,10 +1,12 @@
---
title: "Understand Files"
-weight: 1
-type: docs
-aliases:
-- /learn-paimon/understand-files.html
+sidebar_position: 1
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Label from '@site/src/components/Label';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -39,9 +41,9 @@ we aim to offer insights into the creation and updates of files.
Before delving further into this page, please ensure that you have read through the
following sections:
-1. [Basic Concepts]({{< ref "concepts/basic-concepts" >}}),
-2. [Primary Key Table]({{< ref "primary-key-table/overview" >}}) and [Append Table]({{< ref "append-table/overview" >}})
-3. How to use Paimon in [Flink]({{< ref "flink" >}}).
+1. [Basic Concepts](../concepts/basic-concepts),
+2. [Primary Key Table](../primary-key-table/overview) and [Append Table](../append-table/overview)
+3. How to use Paimon in [Flink](../flink).
## Understand File Operations
@@ -91,7 +93,7 @@ Users can verify the visibility of these records by executing the query `SELECT
The commit process creates a snapshot located at the path `/tmp/paimon/default.db/T/snapshot/snapshot-1`.
The resulting file layout at snapshot-1 is as described below:
-{{< img src="/img/file-operations-0.png">}}
+
The content of snapshot-1 contains metadata of the snapshot, such as manifest list and schema id:
```json
@@ -199,14 +201,14 @@ data-b75b7381-7c8b-430f-b7e5-a204cb65843c-0.orc
schema-0
```
The new file layout as of snapshot-2 looks like
-{{< img src="/img/file-operations-1.png">}}
+
### Delete Records From Table
Now let's delete records that meet the condition `dt>=20230503`.
In Flink SQL, execute the following statement:
-{{< label Batch >}}
+<Label>Batch</Label>
```sql
DELETE FROM T WHERE dt >= '20230503';
```
@@ -229,7 +231,7 @@ the record in the third commit. Executing `SELECT * FROM T` will return 2 rows,
```
The new file layout as of snapshot-3 looks like
-{{< img src="/img/file-operations-2.png">}}
+
Note that `manifest-3-0` contains 8 manifest entries of `ADD` operation type,
corresponding to 8 newly written data files.
@@ -244,11 +246,12 @@ is needed in order to reduce the number of small files.
Let's trigger the full-compaction now, and run a dedicated compaction job through `flink run`:
-{{< label Batch >}}
+<Label>Batch</Label>
-{{< tabs "compact" >}}
+<Tabs groupId="compact">
+
+<TabItem value="flink-sql" label="Flink SQL">
-{{< tab "Flink SQL" >}}
```sql
CALL sys.compact(
`table` => 'database_name.table_name',
@@ -258,13 +261,15 @@ CALL sys.compact(
options => 'paimon_table_dynamic_conf'
);
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
+
```bash
<FLINK_HOME>/bin/flink run \
-D execution.runtime-mode=batch \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -273,32 +278,35 @@ CALL sys.compact(
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]] \
[--table_conf <paimon-table-dynamic-conf> [--table_conf <paimon-table-dynamic-conf>] ...]
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
an example would be (suppose you're already in Flink home)
-{{< tabs "compact example" >}}
+<Tabs groupId="compact-example">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
```sql
CALL sys.compact('T');
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
```bash
./bin/flink run \
- ./lib/paimon-flink-action-{{< version >}}.jar \
+ ./lib/paimon-flink-action-@@VERSION@@.jar \
compact \
--path file:///tmp/paimon/default.db/T
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
All current table files will be compacted and a new snapshot, namely `snapshot-4`, is
made and contains the following information:
@@ -324,7 +332,7 @@ made and contains the following information:
```
The new file layout as of snapshot-4 looks like
-{{< img src="/img/file-operations-3.png">}}
+
Note that `manifest-4-0` contains 20 manifest entries (18 `DELETE` operations and 2 `ADD` operations)
1. For partition `20230503` to `20230510`, two `DELETE` operations for two data files
@@ -344,7 +352,7 @@ has actually used this schema yet until the next commit.
### Expire Snapshots
Remind that the marked data files are not truly deleted until the snapshot expires and
-no consumer depends on the snapshot. For more information, see [Expiring Snapshots]({{< ref "maintenance/manage-snapshots#expiring-snapshots" >}}).
+no consumer depends on the snapshot. For more information, see [Expiring Snapshots](../maintenance/manage-snapshots#expire-snapshots).
During the process of snapshot expiration, the range of snapshots is initially determined, and then data files within these snapshots are marked for deletion.
A data file is `marked` for deletion only when there is a manifest entry of kind `DELETE` that references that specific data file.
@@ -366,7 +374,7 @@ Let's say another snapshot, `snapshot-5` is created and snapshot expiration is t
to be deleted. For simplicity, we will only focus on files from previous snapshots, the final layout after snapshot
expiration looks like:
-{{< img src="/img/file-operations-4.png">}}
+
As a result, partition `20230503` to `20230510` are physically deleted.
@@ -380,7 +388,7 @@ and snapshot commit and expiration.
To begin, let's take a closer look at the CDC ingestion workflow and
the unique roles played by each component involved.
-{{< img src="/img/cdc-ingestion-topology.png">}}
+
1. `MySQL CDC Source` uniformly reads snapshot and incremental data, with `SnapshotReader` reading snapshot data
and `BinlogReader` reading incremental data, respectively.
@@ -391,11 +399,11 @@ the unique roles played by each component involved.
Next, we will go over end-to-end data flow.
-{{< img src="/img/cdc-ingestion-source.png">}}
+
`MySQL Cdc Source` read snapshot and incremental data and emit them to downstream after normalization.
-{{< img src="/img/cdc-ingestion-write.png">}}
+
`Paimon Sink` first buffers new records in a heap-based LSM tree, and flushes them to disk when
@@ -403,12 +411,12 @@ the memory buffer is full. Note that each data file written is a sorted run. At
is created. Right before Flink checkpoint takes places, `Paimon Sink` will flush all buffered records and send committable message
to downstream, which is read and committed by `Committer Operator` during checkpoint.
-{{< img src="/img/cdc-ingestion-commit.png">}}
+
During checkpoint, `Committer Operator` will create a new snapshot and associate it with manifest lists so that the snapshot
contains information about all data files in the table.
-{{< img src="/img/cdc-ingestion-compact.png">}}
+
At later point asynchronous compaction might take place, and the committable produced by `CompactManager` contains information
about previous files and merged files so that `Committer Operator` can construct corresponding manifest entries. In this case
@@ -439,20 +447,20 @@ spilled files in writer to generate bigger files in DFS.
### Understand Snapshots
-{{< img src="/img/file-operations-3.png">}}
+
Paimon maintains multiple versions of files, compaction and deletion of files are logical and do not actually
delete files. Files are only really deleted when Snapshot is expired, so the first way to reduce files is to
reduce the time it takes for snapshot to be expired. Flink writer will automatically expire snapshots.
-See [Expire Snapshots]({{< ref "/maintenance/manage-snapshots#expire-snapshots" >}}).
+See [Expire Snapshots](../maintenance/manage-snapshots#expire-snapshots).
### Understand Partitions and Buckets
Paimon files are organized in a layered style. The following image illustrates the file layout. Starting
from a snapshot file, Paimon readers can recursively access all records from the table.
-{{< img src="/img/file-layout.png">}}
+
For example, the following table:
@@ -478,9 +486,9 @@ of buckets, otherwise there will be quite a few small files as well.
LSM tree organizes files into several sorted runs. A sorted run consists of one or multiple data files and each data
file belongs to exactly one sorted run.
-{{< img src="/img/sorted-runs.png">}}
+
-By default, sorted runs number depends on `num-sorted-run.compaction-trigger`, see [Compaction for Primary Key Table]({{< ref "/maintenance/write-performance#compaction" >}}),
+By default, sorted runs number depends on `num-sorted-run.compaction-trigger`, see [Compaction for Primary Key Table](../primary-key-table/compaction),
this means that there are at least 5 files in a bucket. If you want to reduce this number, you can keep fewer files, but write performance may suffer.
### Understand Files for Bucketed Append Table
@@ -488,7 +496,7 @@ this means that there are at least 5 files in a bucket. If you want to reduce th
By default, Append also does automatic compaction to reduce the number of small files.
However, for Bucketed Append table, it will only compact the files within the Bucket for sequential
-purposes, which may keep more small files. See [Bucketed Append]({{< ref "append-table/bucketed" >}}).
+purposes, which may keep more small files. See [Bucketed Append](../append-table/bucketed).
### Understand Full-Compaction
diff --git a/docs/docs/maintenance/configurations.mdx b/docs/docs/maintenance/configurations.mdx
new file mode 100644
index 0000000000..e2cb83d8d8
--- /dev/null
+++ b/docs/docs/maintenance/configurations.mdx
@@ -0,0 +1,102 @@
+---
+title: "Configurations"
+sidebar_position: 100
+---
+
+import ConfigTable from '@site/src/components/ConfigTable';
+import coreConfigurationHtml from '@site/generated/core_configuration.html';
+import catalogConfigurationHtml from '@site/generated/catalog_configuration.html';
+import hiveCatalogConfigurationHtml from '@site/generated/hive_catalog_configuration.html';
+import hiveConnectorConfigurationHtml from '@site/generated/hive_connector_configuration.html';
+import jdbcCatalogConfigurationHtml from '@site/generated/jdbc_catalog_configuration.html';
+import flinkCatalogConfigurationHtml from '@site/generated/flink_catalog_configuration.html';
+import flinkConnectorConfigurationHtml from '@site/generated/flink_connector_configuration.html';
+import sparkCatalogConfigurationHtml from '@site/generated/spark_catalog_configuration.html';
+import sparkConnectorConfigurationHtml from '@site/generated/spark_connector_configuration.html';
+import orcConfigurationHtml from '@site/generated/orc_configuration.html';
+import rocksdbConfigurationHtml from '@site/generated/rocksdb_configuration.html';
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+### CoreOptions
+
+Core options for paimon.
+
+<ConfigTable html={coreConfigurationHtml} />
+
+### CatalogOptions
+
+Options for paimon catalog.
+
+<ConfigTable html={catalogConfigurationHtml} />
+
+### HiveCatalogOptions
+
+Options for Hive catalog.
+
+<ConfigTable html={hiveCatalogConfigurationHtml} />
+
+### HiveConnectorOptions
+
+Hive connector options for paimon.
+
+<ConfigTable html={hiveConnectorConfigurationHtml} />
+
+### JdbcCatalogOptions
+
+Options for Jdbc catalog.
+
+<ConfigTable html={jdbcCatalogConfigurationHtml} />
+
+### FlinkCatalogOptions
+
+Flink catalog options for paimon.
+
+<ConfigTable html={flinkCatalogConfigurationHtml} />
+
+### FlinkConnectorOptions
+
+Flink connector options for paimon.
+
+<ConfigTable html={flinkConnectorConfigurationHtml} />
+
+### SparkCatalogOptions
+
+Spark catalog options for paimon.
+
+<ConfigTable html={sparkCatalogConfigurationHtml} />
+
+### SparkConnectorOptions
+
+Spark connector options for paimon.
+
+<ConfigTable html={sparkConnectorConfigurationHtml} />
+
+### ORC Options
+
+<ConfigTable html={orcConfigurationHtml} />
+
+### RocksDB Options
+
+The following options allow users to finely adjust RocksDB for better performance. You can either specify them in table properties or in dynamic table hints.
+
+<ConfigTable html={rocksdbConfigurationHtml} />
diff --git a/docs/content/maintenance/dedicated-compaction.md b/docs/docs/maintenance/dedicated-compaction.mdx
similarity index 84%
rename from docs/content/maintenance/dedicated-compaction.md
rename to docs/docs/maintenance/dedicated-compaction.mdx
index f1af09f0ac..ea781e70ac 100644
--- a/docs/content/maintenance/dedicated-compaction.md
+++ b/docs/docs/maintenance/dedicated-compaction.mdx
@@ -1,10 +1,13 @@
---
title: "Dedicated Compaction"
-weight: 4
-type: docs
-aliases:
-- /maintenance/dedicated-compaction.html
+sidebar_position: 4
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import ConfigTable from '@site/src/components/ConfigTable';
+import sortCompactHtml from '@site/generated/sort-compact.html';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -28,16 +31,18 @@ under the License.
Paimon's snapshot management supports writing with multiple writers.
-{{< hint info >}}
+:::info
+
For S3-like object store, its `'RENAME'` does not have atomic semantic. We need to configure Hive metastore and
enable `'lock.enabled'` option for the catalog.
-{{< /hint >}}
+
+:::
By default, Paimon supports concurrent writing to different partitions. A recommended mode is that streaming
job writes records to Paimon's latest partition, Simultaneously batch job (overwrite) writes records to the
historical partition.
-{{< img src="/img/multiple-writers.png">}}
+
So far, everything works very well, but if you need multiple writers to write records to the same partition, it will
be a bit more complicated. For example, you don't want to use `UNION ALL`, you have multiple
@@ -48,7 +53,7 @@ streaming jobs to write records to a `'partial-update'` table. Please refer to t
By default, Paimon writers will perform compaction as needed during writing records. This is sufficient for most use cases.
Compaction will mark some data files as "deleted" (not really deleted, see
-[expiring snapshots]({{< ref "maintenance/manage-snapshots#expiring-snapshots" >}}) for more info).
+[expiring snapshots](./manage-snapshots#expire-snapshots) for more info).
If multiple writers mark the same file, a conflict will occur when committing the changes. Paimon will automatically
resolve the conflict, but this may result in job restarts.
@@ -56,21 +61,21 @@ To avoid these downsides, users can also choose to skip compactions in writers,
To skip compactions in writers, set the following table property to `true`.
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Option</th>
- <th class="text-left" style="width: 5%">Required</th>
- <th class="text-left" style="width: 5%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 60%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Option</th>
+ <th className="text-left" style={{width: "5%"}}>Required</th>
+ <th className="text-left" style={{width: "5%"}}>Default</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "60%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>write-only</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">false</td>
+ <td style={{wordWrap: "break-word"}}>false</td>
<td>Boolean</td>
<td>If set to true, compactions and snapshot expiration will be skipped. This option is used along with dedicated compact jobs.</td>
</tr>
@@ -79,9 +84,9 @@ To skip compactions in writers, set the following table property to `true`.
To run a dedicated job for compaction, follow these instructions.
-{{< tabs "dedicated-compaction-job" >}}
+<Tabs groupId="dedicated-compaction-job">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
@@ -93,15 +98,16 @@ CALL sys.compact(
`where` => 'dt>10 and h<20'
);
```
-{{< /tab >}}
-{{< tab "Flink Action Jar" >}}
+</TabItem>
+
+<TabItem value="flink-action-jar" label="Flink Action Jar">
Run the following command to submit a compaction job for the table.
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -116,7 +122,7 @@ Example: compact table
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse s3:///path/to/warehouse \
--database test_db \
@@ -141,26 +147,28 @@ For more usage of the compact action, see
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact --help
```
-{{< /tab >}}
+</TabItem>
+
+</Tabs>
-{{< /tabs >}}
+:::info
-{{< hint info >}}
Similarly, the default is synchronous compaction, which may cause checkpoint timeouts.
-You can configure `table_conf` to use [Asynchronous Compaction]({{< ref "primary-key-table/compaction#asynchronous-compaction" >}}).
-{{< /hint >}}
+You can configure `table_conf` to use [Asynchronous Compaction](../primary-key-table/compaction#asynchronous-compaction).
+
+:::
## Database Compaction Job
You can run the following command to submit a compaction job for multiple database.
-{{< tabs "database-compaction-job" >}}
+<Tabs groupId="database-compaction-job">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
@@ -182,13 +190,14 @@ CALL sys.compact_database(
table_options => 'sink.parallelism=4'
)
```
-{{< /tab >}}
-{{< tab "Flink Action Jar" >}}
+</TabItem>
+
+<TabItem value="flink-action-jar" label="Flink Action Jar">
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact_database \
--warehouse <warehouse-path> \
--including_databases <database-name|name-regular-expr> \
@@ -206,7 +215,7 @@ CALL sys.compact_database(
* `--mode` is used to specify compaction mode. Possible values:
* "divided" (the default mode if you haven't specified one): start a sink for each table, the compaction of the new table requires restarting the job.
* "combined": start a single combined sink for all tables, the new table will be automatically compacted.
-* `--catalog_conf` is the configuration for Paimon catalog. Each configuration should be specified in the format `key=value`. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of catalog configurations.
+* `--catalog_conf` is the configuration for Paimon catalog. Each configuration should be specified in the format `key=value`. See [here](./configurations) for a complete list of catalog configurations.
* `--table_conf` is the configuration for compaction. Each configuration should be specified in the format `key=value`. Pivotal configuration is listed below:
| Key | Default | Type | Description |
@@ -218,23 +227,23 @@ You can use `-D execution.runtime-mode=batch` to control batch or streaming mode
current table files will be compacted. If you submit a streaming job, the job will continuously monitor new changes
to the table and perform compactions as needed.
-{{< hint info >}}
+:::info
If you only want to submit the compaction job and don't want to wait until the job is done, you should submit in [detached mode](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/cli/#submitting-a-job).
-{{< /hint >}}
+:::
-{{< hint info >}}
+:::info
You can set `--mode combined` to enable compacting newly added tables without restarting job.
-{{< /hint >}}
+:::
Example1: compact database
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact_database \
--warehouse s3:///path/to/warehouse \
--including_databases test_db \
@@ -247,7 +256,7 @@ Example2: compact database in combined mode
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact_database \
--warehouse s3:///path/to/warehouse \
--including_databases test_db \
@@ -262,23 +271,23 @@ For more usage of the compact_database action, see
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact_database --help
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Sort Compact
-If your table is configured with [dynamic bucket primary key table]({{< ref "primary-key-table/data-distribution#dynamic-bucket" >}})
-or [append table]({{< ref "append-table/overview" >}}) ,
+If your table is configured with [dynamic bucket primary key table](../primary-key-table/data-distribution#dynamic-bucket)
+or [append table](../append-table/overview) ,
you can trigger a compact with specified column sort to speed up queries.
-{{< tabs "sort-compaction-job" >}}
+<Tabs groupId="sort-compaction-job">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
@@ -286,14 +295,15 @@ Run the following sql:
-- sort compact table
CALL sys.compact(`table` => 'default.T', order_strategy => 'zorder', order_by => 'a,b')
```
-{{< /tab >}}
-{{< tab "Flink Action Jar" >}}
+</TabItem>
+
+<TabItem value="flink-action-jar" label="Flink Action Jar">
```bash
<FLINK_HOME>/bin/flink run \
-D execution.runtime-mode=batch \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -305,35 +315,37 @@ CALL sys.compact(`table` => 'default.T', order_strategy => 'zorder', order_by =>
[--table_conf <paimon-table-dynamic-conf> [--table_conf <paimon-table-dynamic-conf>] ...]
```
There are two new configuration in `Sort Compact`
-{{< generated/sort-compact >}}
+<ConfigTable html={sortCompactHtml} />
The sort parallelism is the same as the sink parallelism, you can dynamically specify it by add conf `--table_conf sink.parallelism=<value>`.
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
+
+:::info
-{{< hint info >}}
Sort Compact currently supports only bucket=-1 and batch mode.
-{{< /hint >}}
+
+:::
## Historical Partition Compact
You can run the following command to submit a compaction job for partition which has not received any new data for
a period of time. Small files in those partitions will be full compacted.
-{{< hint info >}}
+:::info
This feature now is only used in batch mode.
-{{< /hint >}}
+:::
### For Table
This is for one table.
-{{< tabs "history-partition-compaction-job for table" >}}
+<Tabs groupId="history-partition-compaction-job-for-table">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
@@ -342,14 +354,14 @@ Run the following sql:
CALL sys.compact(`table` => 'default.T', partition_idle_time => '1 d')
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action Jar" >}}
+<TabItem value="flink-action-jar" label="Flink Action Jar">
```bash
<FLINK_HOME>/bin/flink run \
-D execution.runtime-mode=batch \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -366,16 +378,16 @@ There are one new configuration in `Historical Partition Compact`
* `--partition_idle_time`: this is used to do a full compaction for partition which had not received any new data for
'partition_idle_time'. And only these partitions will be compacted.
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
### For Databases
This is for multiple tables in different databases.
-{{< tabs "history-partition-compaction-job for databases" >}}
+<Tabs groupId="history-partition-compaction-job-for-databases">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
@@ -402,14 +414,14 @@ CALL sys.compact_database(
);
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action Jar" >}}
+<TabItem value="flink-action-jar" label="Flink Action Jar">
```bash
<FLINK_HOME>/bin/flink run \
-D execution.runtime-mode=batch \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact_database \
--warehouse <warehouse-path> \
--including_databases <database-name|name-regular-expr> \
@@ -426,7 +438,7 @@ Example: compact historical partitions for tables in database
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
compact_database \
--warehouse s3:///path/to/warehouse \
--including_databases test_db \
@@ -436,6 +448,6 @@ Example: compact historical partitions for tables in database
--catalog_conf s3.secret-key=*****
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
\ No newline at end of file
+</Tabs>
\ No newline at end of file
diff --git a/docs/content/maintenance/filesystems.md b/docs/docs/maintenance/filesystems.mdx
similarity index 70%
rename from docs/content/maintenance/filesystems.md
rename to docs/docs/maintenance/filesystems.mdx
index 38bcc090fe..98a90ec45e 100644
--- a/docs/content/maintenance/filesystems.md
+++ b/docs/docs/maintenance/filesystems.mdx
@@ -1,10 +1,13 @@
---
title: "Filesystems"
-weight: 1
-type: docs
-aliases:
-- /maintenance/filesystems.html
+sidebar_position: 1
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import Stable from '@site/src/components/Stable';
+import Unstable from '@site/src/components/Unstable';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -48,11 +51,11 @@ FileSystem pluggable jars for user to query tables from Spark/Hive side.
## Dependency
-We recommend you to download the jar directly: [Download Link]({{< ref "project/download#filesystem-jars" >}}).
+We recommend you to download the jar directly: [Download Link](../project/download#filesystem-jars).
You can also manually build bundled jar from the source code.
-To build from source code, [clone the git repository]({{< github_repo >}}).
+To build from source code, [clone the git repository](@@GITHUB_REPO@@).
Build shaded jar with the following command.
@@ -61,7 +64,7 @@ mvn clean install -DskipTests
```
You can find the shaded jars under
-`./paimon-filesystems/paimon-${fs}/target/paimon-${fs}-{{< version >}}.jar`.
+`./paimon-filesystems/paimon-${fs}/target/paimon-${fs}-@@VERSION@@.jar`.
## HDFS
@@ -71,9 +74,9 @@ You don't need any additional dependencies to access HDFS because you have alrea
For HDFS, the most important thing is to be able to read your HDFS configuration.
-{{< tabs "hdfs conf" >}}
+<Tabs groupId="hdfs-conf">
-{{< tab "Flink" >}}
+<TabItem value="flink" label="Flink">
You may not have to do anything, if you are in a hadoop environment. Otherwise, pick one of the following ways to
configure your HDFS:
@@ -86,34 +89,42 @@ The first approach is recommended.
If you do not want to include the value of the environment variable, you can configure `hadoop-conf-loader` to `option`.
-{{< /tab >}}
+</TabItem>
-{{< tab "Hive/Spark" >}}
+<TabItem value="hive-spark" label="Hive/Spark">
HDFS Configuration is available directly through the computation cluster, see cluster configuration of Hive and Spark for details.
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
### Paimon-Hadoop Uber package
If you want to read from and write to Paimon tables on a Hadoop cluster within an application that has no Hadoop dependencies,
you can use the Paimon Hadoop uber jar.
-{{< stable >}}
+<Stable>
+
+
+
+Download [paimon-hadoop-uber-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hadoop-uber/@@VERSION@@/paimon-hadoop-uber-@@VERSION@@.jar).
+
-Download [paimon-hadoop-uber-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-hadoop-uber/{{< version >}}/paimon-hadoop-uber-{{< version >}}.jar).
-{{< /stable >}}
+</Stable>
-{{< unstable >}}
+<Unstable>
-Download [paimon-hadoop-uber-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hadoop-uber/{{< version >}}/).
-{{< /unstable >}}
-Put `paimon-hadoop-uber-{{< version >}}.jar` into classpath directory of your application.
+Download [paimon-hadoop-uber-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-hadoop-uber/@@VERSION@@/).
+
+
+
+</Unstable>
+
+Put `paimon-hadoop-uber-@@VERSION@@.jar` into classpath directory of your application.
### Hadoop-compatible file systems (HCFS)
@@ -140,9 +151,9 @@ For Alluxio support add the following entry into the core-site.xml file:
### Kerberos
-{{< tabs "Kerberos" >}}
+<Tabs groupId="kerberos">
-{{< tab "Flink" >}}
+<TabItem value="flink" label="Flink">
Configure the following options in your catalog configuration:
@@ -155,21 +166,21 @@ And configure the following option in the program's java property:
- java.security.krb5.conf: Absolute path to the Kerberos configuration file.
Please make sure it is copied to each machine.
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark" >}}
+<TabItem value="spark" label="Spark">
It is recommended to use [Spark Kerberos Keytab](https://spark.apache.org/docs/latest/security.html#using-a-keytab).
-{{< /tab >}}
+</TabItem>
-{{< tab "Hive" >}}
+<TabItem value="hive" label="Hive">
An intuitive approach is to configure Hive's kerberos authentication.
-{{< /tab >}}
+</TabItem>
-{{< tab "Trino/JavaAPI" >}}
+<TabItem value="trino-javaapi" label="Trino/JavaAPI">
Configure the following three options in your catalog configuration:
@@ -183,9 +194,9 @@ For JavaAPI:
SecurityContext.install(catalogOptions);
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
### HDFS HA
@@ -197,28 +208,38 @@ Ensure that `hdfs-site.xml` and `core-site.xml` contain the necessary [ViewFs co
## OSS
-{{< stable >}}
+<Stable>
+
-Download [paimon-oss-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/{{< version >}}/paimon-oss-{{< version >}}.jar).
-{{< /stable >}}
+Download [paimon-oss-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-oss/@@VERSION@@/paimon-oss-@@VERSION@@.jar).
-{{< unstable >}}
-Download [paimon-oss-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-oss/{{< version >}}/).
-{{< /unstable >}}
+</Stable>
-{{< tabs "oss" >}}
+<Unstable>
-{{< tab "Flink" >}}
-{{< hint info >}}
+
+Download [paimon-oss-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-oss/@@VERSION@@/).
+
+
+
+</Unstable>
+
+<Tabs groupId="oss">
+
+<TabItem value="flink" label="Flink">
+
+:::info
+
If you have already configured [oss access through Flink](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/oss/) (Via Flink FileSystem),
here you can skip the following configuration.
-{{< /hint >}}
-Put `paimon-oss-{{< version >}}.jar` into `lib` directory of your Flink home, and create catalog:
+:::
+
+Put `paimon-oss-@@VERSION@@.jar` into `lib` directory of your Flink home, and create catalog:
```sql
CREATE CATALOG my_catalog WITH (
@@ -230,15 +251,17 @@ CREATE CATALOG my_catalog WITH (
);
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="spark" label="Spark">
-{{< tab "Spark" >}}
+:::info
-{{< hint info >}}
If you have already configured oss access through Spark (Via Hadoop FileSystem), here you can skip the following configuration.
-{{< /hint >}}
-Place `paimon-oss-{{< version >}}.jar` together with `paimon-spark-{{< version >}}.jar` under Spark's jars directory, and start like
+:::
+
+Place `paimon-oss-@@VERSION@@.jar` together with `paimon-spark-@@VERSION@@.jar` under Spark's jars directory, and start like
```shell
spark-sql \
@@ -250,17 +273,19 @@ spark-sql \
--conf spark.sql.catalog.paimon.fs.oss.accessKeySecret=yyy
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="hive" label="Hive">
-{{< tab "Hive" >}}
+:::info
-{{< hint info >}}
If you have already configured oss access through Hive (Via Hadoop FileSystem), here you can skip the following configuration.
-{{< /hint >}}
+
+:::
NOTE: You need to ensure that Hive metastore can access `oss`.
-Place `paimon-oss-{{< version >}}.jar` together with `paimon-hive-connector-{{< version >}}.jar` under Hive's auxlib directory, and start like
+Place `paimon-oss-@@VERSION@@.jar` together with `paimon-hive-connector-@@VERSION@@.jar` under Hive's auxlib directory, and start like
```sql
SET paimon.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com;
@@ -268,60 +293,79 @@ SET paimon.fs.oss.accessKeyId=xxx;
SET paimon.fs.oss.accessKeySecret=yyy;
```
-And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore]({{< ref "flink/sql-ddl" >}})
+And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore](../flink/sql-ddl)
```sql
SELECT * FROM test_table;
SELECT COUNT(1) FROM test_table;
```
+</TabItem>
-{{< /tab >}}
-{{< tab "Trino" >}}
+<TabItem value="trino" label="Trino">
From version 0.8, paimon-trino uses trino filesystem as basic file read and write system. We strongly recommend you to use jindo-sdk in trino.
-You can find [How to config jindo sdk on trino](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto.md) here.
+You can find [How to config jindo sdk on trino](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto) here.
Please note that:
* Use paimon to replace hive-hadoop2 when you decompress the plugin jar and find location to put in.
* You can specify the `core-site.xml` in `paimon.properties` on configuration [hive.config.resources](https://trino.io/docs/current/connector/hive.html#hdfs-configuration).
* Presto and Jindo use the same configuration method.
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
If your environment has jindo sdk dependencies, you can use Jindo Fs to connect OSS. Jindo has better read and write efficiency.
-{{< stable >}}
-Download [paimon-jindo-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-jindo/{{< version >}}/paimon-jindo-{{< version >}}.jar).
-{{< /stable >}}
-{{< unstable >}}
-Download [paimon-jindo-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-jindo/{{< version >}}/).
-{{< /unstable >}}
+<Stable>
+
+
+Download [paimon-jindo-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-jindo/@@VERSION@@/paimon-jindo-@@VERSION@@.jar).
+
+
+</Stable>
+<Unstable>
+
+
+Download [paimon-jindo-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-jindo/@@VERSION@@/).
+
+
+</Unstable>
## S3
-{{< stable >}}
+<Stable>
+
+
+
+Download [paimon-s3-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/@@VERSION@@/paimon-s3-@@VERSION@@.jar).
+
+
+
+</Stable>
+
+<Unstable>
+
+
-Download [paimon-s3-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/{{< version >}}/paimon-s3-{{< version >}}.jar).
+Download [paimon-s3-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-s3/@@VERSION@@/).
-{{< /stable >}}
-{{< unstable >}}
-Download [paimon-s3-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-s3/{{< version >}}/).
+</Unstable>
-{{< /unstable >}}
+<Tabs groupId="s3">
-{{< tabs "s3" >}}
+<TabItem value="flink" label="Flink">
-{{< tab "Flink" >}}
+:::info
-{{< hint info >}}
If you have already configured [s3 access through Flink](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/s3/) (Via Flink FileSystem),
here you can skip the following configuration.
-{{< /hint >}}
-Put `paimon-s3-{{< version >}}.jar` into `lib` directory of your Flink home, and create catalog:
+:::
+
+Put `paimon-s3-@@VERSION@@.jar` into `lib` directory of your Flink home, and create catalog:
```sql
CREATE CATALOG my_catalog WITH (
@@ -333,15 +377,17 @@ CREATE CATALOG my_catalog WITH (
);
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="spark" label="Spark">
-{{< tab "Spark" >}}
+:::info
-{{< hint info >}}
If you have already configured s3 access through Spark (Via Hadoop FileSystem), here you can skip the following configuration.
-{{< /hint >}}
-Place `paimon-s3-{{< version >}}.jar` together with `paimon-spark-{{< version >}}.jar` under Spark's jars directory, and start like
+:::
+
+Place `paimon-s3-@@VERSION@@.jar` together with `paimon-spark-@@VERSION@@.jar` under Spark's jars directory, and start like
```shell
spark-sql \
@@ -353,17 +399,19 @@ spark-sql \
--conf spark.sql.catalog.paimon.s3.secret-key=yyy
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Hive" >}}
+<TabItem value="hive" label="Hive">
+
+:::info
-{{< hint info >}}
If you have already configured s3 access through Hive ((Via Hadoop FileSystem)), here you can skip the following configuration.
-{{< /hint >}}
+
+:::
NOTE: You need to ensure that Hive metastore can access `s3`.
-Place `paimon-s3-{{< version >}}.jar` together with `paimon-hive-connector-{{< version >}}.jar` under Hive's auxlib directory, and start like
+Place `paimon-s3-@@VERSION@@.jar` together with `paimon-hive-connector-@@VERSION@@.jar` under Hive's auxlib directory, and start like
```sql
SET paimon.s3.endpoint=your-endpoint-hostname;
@@ -371,23 +419,23 @@ SET paimon.s3.access-key=xxx;
SET paimon.s3.secret-key=yyy;
```
-And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore]({{< ref "flink/sql-ddl" >}})
+And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore](../flink/sql-ddl)
```sql
SELECT * FROM test_table;
SELECT COUNT(1) FROM test_table;
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Trino" >}}
+<TabItem value="trino" label="Trino">
Paimon use shared trino filesystem as basic read and write system.
Please refer to [Trino S3](https://trino.io/docs/current/object-storage/file-system-s3.html) to config s3 filesystem in trino.
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
### S3 Compliant Object Stores
@@ -419,28 +467,38 @@ Try to configure this in catalog options: `fs.s3a.connection.maximum=1000`.
## Google Cloud Storage
-{{< stable >}}
+<Stable>
+
+
+
+Download [paimon-gs-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-gs/@@VERSION@@/paimon-gs-@@VERSION@@.jar).
+
+
+
+</Stable>
-Download [paimon-gs-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-gs/{{< version >}}/paimon-gs-{{< version >}}.jar).
+<Unstable>
-{{< /stable >}}
-{{< unstable >}}
-Download [paimon-gs-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-gs/{{< version >}}/).
+Download [paimon-gs-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-gs/@@VERSION@@/).
-{{< /unstable >}}
-{{< tabs "gs" >}}
-{{< tab "Flink" >}}
+</Unstable>
+
+<Tabs groupId="gs">
+
+<TabItem value="flink" label="Flink">
+
+:::info
-{{< hint info >}}
If you have already configured [gcs access through Flink](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/filesystems/gcs/) (Via Flink FileSystem),
here you can skip the following configuration.
-{{< /hint >}}
-Put `paimon-gs-{{< version >}}.jar` into `lib` directory of your Flink home, and create catalog:
+:::
+
+Put `paimon-gs-@@VERSION@@.jar` into `lib` directory of your Flink home, and create catalog:
```sql
CREATE CATALOG my_catalog WITH (
@@ -451,34 +509,44 @@ CREATE CATALOG my_catalog WITH (
);
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Microsoft Azure Storage
-{{< stable >}}
+<Stable>
+
+
+
+Download [paimon-azure-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-azure/@@VERSION@@/paimon-azure-@@VERSION@@.jar).
+
+
+
+</Stable>
-Download [paimon-azure-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-azure/{{< version >}}/paimon-azure-{{< version >}}.jar).
+<Unstable>
-{{< /stable >}}
-{{< unstable >}}
-Download [paimon-azure-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-azure/{{< version >}}/).
+Download [paimon-azure-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-azure/@@VERSION@@/).
-{{< /unstable >}}
-{{< tabs "azure" >}}
-{{< tab "Flink" >}}
+</Unstable>
+
+<Tabs groupId="azure">
+
+<TabItem value="flink" label="Flink">
+
+:::info
-{{< hint info >}}
If you have already configured [azure access through Flink](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/filesystems/azure/) (Via Flink FileSystem),
here you can skip the following configuration.
-{{< /hint >}}
-Put `paimon-azure-{{< version >}}.jar` into `lib` directory of your Flink home, and create catalog:
+:::
+
+Put `paimon-azure-@@VERSION@@.jar` into `lib` directory of your Flink home, and create catalog:
```sql
CREATE CATALOG my_catalog WITH (
@@ -488,15 +556,17 @@ CREATE CATALOG my_catalog WITH (
);
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="spark" label="Spark">
-{{< tab "Spark" >}}
+:::info
-{{< hint info >}}
If you have already configured azure access through Spark (Via Hadoop FileSystem), here you can skip the following configuration.
-{{< /hint >}}
-Place `paimon-azure-{{< version >}}.jar` together with `paimon-spark-{{< version >}}.jar` under Spark's jars directory, and start like
+:::
+
+Place `paimon-azure-@@VERSION@@.jar` together with `paimon-spark-@@VERSION@@.jar` under Spark's jars directory, and start like
```shell
spark-sql \
@@ -506,34 +576,44 @@ spark-sql \
--conf fs.azure.account.key.Account.blob.core.windows.net=yyy \
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## OBS
-{{< stable >}}
+<Stable>
+
+
+
+Download [paimon-obs-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-obs/@@VERSION@@/paimon-obs-@@VERSION@@.jar).
+
-Download [paimon-obs-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-obs/{{< version >}}/paimon-obs-{{< version >}}.jar).
-{{< /stable >}}
+</Stable>
-{{< unstable >}}
+<Unstable>
-Download [paimon-obs-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-obs/{{< version >}}/).
-{{< /unstable >}}
-{{< tabs "obs" >}}
+Download [paimon-obs-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-obs/@@VERSION@@/).
-{{< tab "Flink" >}}
-{{< hint info >}}
+
+</Unstable>
+
+<Tabs groupId="obs">
+
+<TabItem value="flink" label="Flink">
+
+:::info
+
If you have already configured [obs access through Flink](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/s3/) (Via Flink FileSystem),
here you can skip the following configuration.
-{{< /hint >}}
-Put `paimon-obs-{{< version >}}.jar` into `lib` directory of your Flink home, and create catalog:
+:::
+
+Put `paimon-obs-@@VERSION@@.jar` into `lib` directory of your Flink home, and create catalog:
```sql
CREATE CATALOG my_catalog WITH (
@@ -545,15 +625,17 @@ CREATE CATALOG my_catalog WITH (
);
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="spark" label="Spark">
-{{< tab "Spark" >}}
+:::info
-{{< hint info >}}
If you have already configured obs access through Spark (Via Hadoop FileSystem), here you can skip the following configuration.
-{{< /hint >}}
-Place `paimon-obs-{{< version >}}.jar` together with `paimon-spark-{{< version >}}.jar` under Spark's jars directory, and start like
+:::
+
+Place `paimon-obs-@@VERSION@@.jar` together with `paimon-spark-@@VERSION@@.jar` under Spark's jars directory, and start like
```shell
spark-sql \
@@ -565,17 +647,19 @@ spark-sql \
--conf spark.sql.catalog.paimon.fs.obs.secret.key=yyy
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Hive" >}}
+<TabItem value="hive" label="Hive">
+
+:::info
-{{< hint info >}}
If you have already configured obs access through Hive ((Via Hadoop FileSystem)), here you can skip the following configuration.
-{{< /hint >}}
+
+:::
NOTE: You need to ensure that Hive metastore can access `obs`.
-Place `paimon-obs-{{< version >}}.jar` together with `paimon-hive-connector-{{< version >}}.jar` under Hive's auxlib directory, and start like
+Place `paimon-obs-@@VERSION@@.jar` together with `paimon-hive-connector-@@VERSION@@.jar` under Hive's auxlib directory, and start like
```sql
SET paimon.fs.obs.endpoint=obs-endpoint-hostname;
@@ -583,12 +667,12 @@ SET paimon.fs.obs.access.key=xxx;
SET paimon.fs.obs.secret.key=yyy;
```
-And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore]({{< ref "flink/sql-ddl" >}})
+And read table from hive metastore, table can be created by Flink or Spark, see [Catalog with Hive Metastore](../flink/sql-ddl)
```sql
SELECT * FROM test_table;
SELECT COUNT(1) FROM test_table;
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/maintenance/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/maintenance/index.md
index eacaa231ca..21f5d313d4 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/maintenance/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Maintenance"
+sidebar_position: 94
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/maintenance/manage-branches.md b/docs/docs/maintenance/manage-branches.mdx
similarity index 84%
rename from docs/content/maintenance/manage-branches.md
rename to docs/docs/maintenance/manage-branches.mdx
index ba2c2bf708..3fb79d950b 100644
--- a/docs/content/maintenance/manage-branches.md
+++ b/docs/docs/maintenance/manage-branches.mdx
@@ -1,10 +1,11 @@
---
title: "Manage Branches"
-weight: 11
-type: docs
-aliases:
-- /maintenance/manage-branches.html
+sidebar_position: 11
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -36,9 +37,9 @@ By merge or replace branch operations, users can complete the correcting of data
Paimon supports creating branch from a specific tag, or just creating an empty branch which means the initial state of the created branch is like an empty table.
-{{< tabs "create-branches" >}}
+<Tabs groupId="create-branches">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
@@ -49,15 +50,16 @@ CALL sys.create_branch('default.T', 'branch1', 'tag1');
-- create empty branch named 'branch1'
CALL sys.create_branch('default.T', 'branch1');
```
-{{< /tab >}}
-{{< tab "Flink Action Jar" >}}
+</TabItem>
+
+<TabItem value="flink-action-jar" label="Flink Action Jar">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
create_branch \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -66,9 +68,10 @@ Run the following command:
[--tag_name <tag-name>] \
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
Run the following sql:
@@ -79,36 +82,40 @@ CALL sys.create_branch('default.T', 'branch1', 'tag1');
-- create empty branch named 'branch1'
CALL sys.create_branch('default.T', 'branch1');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Delete Branches
You can delete branch by its name.
-{{< hint warning >}}
-__Note:__ The `Delete Branches` operation only deletes the metadata file. If you want to clear the data written during the branch, use [remove_orphan_files]({{< ref "flink/procedures" >}})
-{{< /hint >}}
+:::warning
-{{< tabs "delete-branches" >}}
+__Note:__ The `Delete Branches` operation only deletes the metadata file. If you want to clear the data written during the branch, use [remove_orphan_files](../flink/procedures)
-{{< tab "Flink SQL" >}}
+:::
+
+<Tabs groupId="delete-branches">
+
+<TabItem value="flink-sql" label="Flink SQL">
Run the following sql:
```sql
CALL sys.delete_branch('default.T', 'branch1');
```
-{{< /tab >}}
-{{< tab "Flink Action Jar" >}}
+</TabItem>
+
+<TabItem value="flink-action-jar" label="Flink Action Jar">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
delete_branch \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -116,26 +123,28 @@ Run the following command:
--branch_name <branch-name> \
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
-{{< tab "Spark SQL" >}}
+</TabItem>
+
+<TabItem value="spark-sql" label="Spark SQL">
Run the following sql:
```sql
CALL sys.delete_branch('default.T', 'branch1');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Read / Write With Branch
You can read or write with branch as below.
-{{< tabs "read-write-branch" >}}
+<Tabs groupId="read-write-branch">
-{{< tab "Flink" >}}
+<TabItem value="flink" label="Flink">
```sql
-- read from branch 'branch1'
@@ -146,9 +155,9 @@ SELECT * FROM `t$branch_branch1` /*+ OPTIONS('consumer-id' = 'myid') */;
INSERT INTO `t$branch_branch1` SELECT ...
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark SQL" >}}
+<TabItem value="spark-sql" label="Spark SQL">
```sql
-- read from branch 'branch1'
@@ -158,18 +167,18 @@ SELECT * FROM `t$branch_branch1`;
INSERT INTO `t$branch_branch1` SELECT ...
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark DataFrame" >}}
+<TabItem value="spark-dataframe" label="Spark DataFrame">
```sql
-- read from branch 'branch1'
spark.read.format("paimon").option("branch", "branch1").table("t")
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Fast Forward
@@ -178,23 +187,23 @@ Fast-Forward the custom branch to main will delete all the snapshots, tags and s
If your branch modifies the schema, after Fast Forward, if it is Spark SQL, you can execute `REFRESH TABLE my_table`
to clean up the cache to avoid inconsistencies caused by caching.
-{{< tabs "fast_forward" >}}
+<Tabs groupId="fast-forward">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
```sql
CALL sys.fast_forward('default.T', 'branch1');
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action Jar" >}}
+<TabItem value="flink-action-jar" label="Flink Action Jar">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
fast_forward \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -203,9 +212,9 @@ Run the following command:
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark SQL" >}}
+<TabItem value="spark-sql" label="Spark SQL">
Run the following sql:
@@ -213,9 +222,9 @@ Run the following sql:
CALL sys.fast_forward('default.T', 'branch1');
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Batch Reading from Fallback Branch
@@ -236,9 +245,9 @@ In this case, you can create a branch for streaming job, and set `scan.fallback-
Let's look at an example.
-{{< tabs "read-fallback-branch" >}}
+<Tabs groupId="read-fallback-branch">
-{{< tab "Flink" >}}
+<TabItem value="flink" label="Flink">
```sql
-- create Paimon table
@@ -296,6 +305,6 @@ SELECT * FROM T;
*/
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
diff --git a/docs/content/maintenance/manage-partitions.md b/docs/docs/maintenance/manage-partitions.md
similarity index 97%
rename from docs/content/maintenance/manage-partitions.md
rename to docs/docs/maintenance/manage-partitions.md
index 09b48899a7..1570cdf8d2 100644
--- a/docs/content/maintenance/manage-partitions.md
+++ b/docs/docs/maintenance/manage-partitions.md
@@ -1,10 +1,8 @@
---
title: "Manage Partitions"
-weight: 12
-type: docs
-aliases:
-- /maintenance/manage-partitions.html
+sidebar_position: 12
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -45,11 +43,13 @@ What is the scenario for this strategy:
- You only want to keep data that has been updated in the last n days/months/years.
- Data initialization imports a large amount of historical data.
-{{< hint info >}}
+:::info
+
__Note:__ After the partition expires, it is logically deleted and the latest snapshot cannot query its data. But the
files in the file system are not immediately physically deleted, it depends on when the corresponding snapshot expires.
-See [Expire Snapshots]({{< ref "/maintenance/manage-snapshots#expire-snapshots" >}}).
-{{< /hint >}}
+See [Expire Snapshots](./manage-snapshots#expire-snapshots).
+
+:::
An example for single partition field:
diff --git a/docs/content/maintenance/manage-privileges.md b/docs/docs/maintenance/manage-privileges.mdx
similarity index 92%
rename from docs/content/maintenance/manage-privileges.md
rename to docs/docs/maintenance/manage-privileges.mdx
index bf07e14b77..2191b559a4 100644
--- a/docs/content/maintenance/manage-privileges.md
+++ b/docs/docs/maintenance/manage-privileges.mdx
@@ -1,10 +1,11 @@
---
title: "Manage Privileges"
-weight: 10
-type: docs
-aliases:
-- /maintenance/manage-privileges.html
+sidebar_position: 10
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -33,12 +34,14 @@ so that you can manage table access in a fine-grained manner.
Currently, Paimon adopts the identity-based access control (IBAC) privilege model.
That is, privileges are directly assigned to users.
-{{< hint warning >}}
+:::warning
+
This privilege system only prevents unwanted users from accessing tables through catalogs.
It does not block access through temporary table (by specifying table path on filesystem),
nor does it prevent user from directly modifying data files on filesystem.
If you need more serious protection, use a filesystem with access management instead.
-{{< /hint >}}
+
+:::
## Basic Concepts
@@ -98,9 +101,10 @@ Only catalogs with `'metastore' = 'filesystem'` (the default value) or `'metasto
To enable the privilege system on a filesystem / Hive catalog, do the following steps.
-{{< tabs "enable-privileges" >}}
+<Tabs groupId="enable-privileges">
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
-{{< tab "Flink 1.18+" >}}
Run the following Flink SQL.
```sql
-- use the catalog where you want to enable the privilege system
@@ -110,19 +114,22 @@ USE CATALOG `my-catalog`;
-- change 'root-password' to the password you want
CALL sys.init_file_based_privilege('root-password');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
After the privilege system is enabled,
please re-create the catalog and authenticate as `root` to create other users and grant them privileges.
-{{< hint info >}}
+:::info
+
Privilege system does not affect existing catalogs.
That is, these catalogs can still access and modify the tables freely.
Please drop and re-create all catalogs with the desired warehouse path
if you want to use the privilege system in these catalogs.
-{{< /hint >}}
+
+:::
## Accessing Privileged Catalogs
@@ -131,9 +138,10 @@ you need to define `user` and `password` catalog options when creating the catal
For example, the following SQL creates a catalog while trying to be authenticated as `root`,
whose password is `mypassword`.
-{{< tabs "access-catalog" >}}
+<Tabs groupId="access-catalog">
+
+<TabItem value="flink" label="Flink">
-{{< tab "Flink" >}}
```sql
CREATE CATALOG `my-catalog` WITH (
'type' = 'paimon',
@@ -142,9 +150,10 @@ CREATE CATALOG `my-catalog` WITH (
'password' = 'mypassword'
);
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Creating Users
@@ -152,9 +161,10 @@ You must be authenticated as a user with `ADMIN` privilege (for example, `root`)
Do the following steps to create a user in the privilege system.
-{{< tabs "create-users" >}}
+<Tabs groupId="create-users">
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
-{{< tab "Flink 1.18+" >}}
Run the following Flink SQL.
```sql
-- use the catalog where you want to create a user
@@ -165,9 +175,10 @@ USE CATALOG `my-catalog`;
-- change 'user' and 'password' to the username and password you want
CALL sys.create_privileged_user('user', 'password');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Dropping Users
@@ -175,9 +186,10 @@ You must be authenticated as a user with `ADMIN` privilege (for example, `root`)
Do the following steps to drop a user in the privilege system.
-{{< tabs "drop-users" >}}
+<Tabs groupId="drop-users">
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
-{{< tab "Flink 1.18+" >}}
Run the following Flink SQL.
```sql
-- use the catalog where you want to drop a user
@@ -187,9 +199,10 @@ USE CATALOG `my-catalog`;
-- change 'user' to the username you want to drop
CALL sys.drop_privileged_user('user');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Granting Privileges to Users
@@ -197,9 +210,10 @@ You must be authenticated as a user with `ADMIN` privilege (for example, `root`)
Do the following steps to grant a user with privilege in the privilege system.
-{{< tabs "grant-to-users" >}}
+<Tabs groupId="grant-to-users">
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
-{{< tab "Flink 1.18+" >}}
Run the following Flink SQL.
```sql
-- use the catalog where you want to drop a user
@@ -214,9 +228,10 @@ CALL sys.grant_privilege_to_user('user', 'SELECT', 'my_db');
-- grant 'user' with privilege 'SELECT' on table my_db.my_tbl
CALL sys.grant_privilege_to_user('user', 'SELECT', 'my_db', 'my_tbl');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
## Revoking Privileges to Users
@@ -224,9 +239,10 @@ You must be authenticated as a user with `ADMIN` privilege (for example, `root`)
Do the following steps to revoke a privilege from user in the privilege system.
-{{< tabs "revoke-from-users" >}}
+<Tabs groupId="revoke-from-users">
+
+<TabItem value="flink-1-18" label="Flink 1.18+">
-{{< tab "Flink 1.18+" >}}
Run the following Flink SQL.
```sql
-- use the catalog where you want to drop a user
@@ -241,6 +257,7 @@ CALL sys.revoke_privilege_from_user('user', 'SELECT', 'my_db');
-- revoke 'user' with privilege 'SELECT' on table my_db.my_tbl
CALL sys.revoke_privilege_from_user('user', 'SELECT', 'my_db', 'my_tbl');
```
-{{< /tab >}}
-{{< /tabs >}}
+</TabItem>
+
+</Tabs>
diff --git a/docs/content/maintenance/manage-snapshots.md b/docs/docs/maintenance/manage-snapshots.mdx
similarity index 73%
rename from docs/content/maintenance/manage-snapshots.md
rename to docs/docs/maintenance/manage-snapshots.mdx
index 242c966d99..821aa0b755 100644
--- a/docs/content/maintenance/manage-snapshots.md
+++ b/docs/docs/maintenance/manage-snapshots.mdx
@@ -1,10 +1,11 @@
---
title: "Manage Snapshots"
-weight: 5
-type: docs
-aliases:
-- /maintenance/manage-snapshots.html
+sidebar_position: 5
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -30,55 +31,55 @@ This section will describe the management and behavior related to snapshots.
## Expire Snapshots
-Paimon writers generate one or two [snapshot]({{< ref "concepts/basic-concepts#snapshot" >}}) per commit. Each snapshot may add some new data files or mark some old data files as deleted. However, the marked data files are not truly deleted because Paimon also supports time traveling to an earlier snapshot. They are only deleted when the snapshot expires.
+Paimon writers generate one or two [snapshot](../concepts/basic-concepts#snapshot) per commit. Each snapshot may add some new data files or mark some old data files as deleted. However, the marked data files are not truly deleted because Paimon also supports time traveling to an earlier snapshot. They are only deleted when the snapshot expires.
Currently, expiration is automatically performed by Paimon writers when committing new changes. By expiring old snapshots, old data files and metadata files that are no longer used can be deleted to release disk space.
Snapshot expiration is controlled by the following table properties.
-<table class="table table-bordered">
+<table className="table table-bordered">
<thead>
<tr>
- <th class="text-left" style="width: 20%">Option</th>
- <th class="text-left" style="width: 5%">Required</th>
- <th class="text-left" style="width: 5%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 60%">Description</th>
+ <th className="text-left" style={{width: "20%"}}>Option</th>
+ <th className="text-left" style={{width: "5%"}}>Required</th>
+ <th className="text-left" style={{width: "5%"}}>Default</th>
+ <th className="text-left" style={{width: "10%"}}>Type</th>
+ <th className="text-left" style={{width: "60%"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>snapshot.time-retained</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">1 h</td>
+ <td style={{wordWrap: "break-word"}}>1 h</td>
<td>Duration</td>
<td>The maximum time of completed snapshots to retain.</td>
</tr>
<tr>
<td><h5>snapshot.num-retained.min</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">10</td>
+ <td style={{wordWrap: "break-word"}}>10</td>
<td>Integer</td>
<td>The minimum number of completed snapshots to retain. Should be greater than or equal to 1.</td>
</tr>
<tr>
<td><h5>snapshot.num-retained.max</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">Integer.MAX_VALUE</td>
+ <td style={{wordWrap: "break-word"}}>Integer.MAX_VALUE</td>
<td>Integer</td>
<td>The maximum number of completed snapshots to retain. Should be greater than or equal to the minimum number.</td>
</tr>
<tr>
<td><h5>snapshot.expire.execution-mode</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">sync</td>
+ <td style={{wordWrap: "break-word"}}>sync</td>
<td>Enum</td>
<td>Specifies the execution mode of expire.</td>
</tr>
<tr>
<td><h5>snapshot.expire.limit</h5></td>
<td>No</td>
- <td style="word-wrap: break-word;">10</td>
+ <td style={{wordWrap: "break-word"}}>10</td>
<td>Integer</td>
<td>The maximum number of snapshots allowed to expire at a time.</td>
</tr>
@@ -94,9 +95,9 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
<table>
<thead>
<tr>
- <th class="text-left" style="width: 30%">New Snapshots</th>
- <th class="text-left" style="width: 40%">All snapshots after expiration check</th>
- <th class="text-left" style="width: 30%">explanation</th>
+ <th className="text-left" style={{width: "30%"}}>New Snapshots</th>
+ <th className="text-left" style={{width: "40%"}}>All snapshots after expiration check</th>
+ <th className="text-left" style={{width: "30%"}}>explanation</th>
</tr>
</thead>
<tbody>
@@ -116,7 +117,7 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-2, 2023-07-06 10:20)
</td>
<td>
- (snapshots-1, 2023-07-06 10:00) <br>
+ (snapshots-1, 2023-07-06 10:00) <br/>
(snapshots-2, 2023-07-06 10:20)
</td>
<td>
@@ -128,8 +129,8 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-3, 2023-07-06 10:40)
</td>
<td>
- (snapshots-1, 2023-07-06 10:00) <br>
- (snapshots-2, 2023-07-06 10:20) <br>
+ (snapshots-1, 2023-07-06 10:00) <br/>
+ (snapshots-2, 2023-07-06 10:20) <br/>
(snapshots-3, 2023-07-06 10:40)
</td>
<td>
@@ -141,10 +142,10 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-4, 2023-07-06 11:00)
</td>
<td>
- (snapshots-1, 2023-07-06 10:00) <br>
- (snapshots-2, 2023-07-06 10:20) <br>
- (snapshots-3, 2023-07-06 10:40) <br>
- (snapshots-4, 2023-07-06 11:00) <br>
+ (snapshots-1, 2023-07-06 10:00) <br/>
+ (snapshots-2, 2023-07-06 10:20) <br/>
+ (snapshots-3, 2023-07-06 10:40) <br/>
+ (snapshots-4, 2023-07-06 11:00) <br/>
</td>
<td>
No snapshot expired
@@ -155,9 +156,9 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-5, 2023-07-06 11:20)
</td>
<td>
- (snapshots-2, 2023-07-06 10:20) <br>
- (snapshots-3, 2023-07-06 10:40) <br>
- (snapshots-4, 2023-07-06 11:00) <br>
+ (snapshots-2, 2023-07-06 10:20) <br/>
+ (snapshots-3, 2023-07-06 10:40) <br/>
+ (snapshots-4, 2023-07-06 11:00) <br/>
(snapshots-5, 2023-07-06 11:20)
</td>
<td>
@@ -169,9 +170,9 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-6, 2023-07-06 11:30)
</td>
<td>
- (snapshots-3, 2023-07-06 10:40) <br>
- (snapshots-4, 2023-07-06 11:00) <br>
- (snapshots-5, 2023-07-06 11:20) <br>
+ (snapshots-3, 2023-07-06 10:40) <br/>
+ (snapshots-4, 2023-07-06 11:00) <br/>
+ (snapshots-5, 2023-07-06 11:20) <br/>
(snapshots-6, 2023-07-06 11:30)
</td>
<td>
@@ -183,10 +184,10 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-7, 2023-07-06 11:35)
</td>
<td>
- (snapshots-3, 2023-07-06 10:40) <br>
- (snapshots-4, 2023-07-06 11:00) <br>
- (snapshots-5, 2023-07-06 11:20) <br>
- (snapshots-6, 2023-07-06 11:30) <br>
+ (snapshots-3, 2023-07-06 10:40) <br/>
+ (snapshots-4, 2023-07-06 11:00) <br/>
+ (snapshots-5, 2023-07-06 11:20) <br/>
+ (snapshots-6, 2023-07-06 11:30) <br/>
(snapshots-7, 2023-07-06 11:35)
</td>
<td>
@@ -198,10 +199,10 @@ The following example show more details(`snapshot.num-retained.min` is 2, `snaps
(snapshots-8, 2023-07-06 11:36)
</td>
<td>
- (snapshots-4, 2023-07-06 11:00) <br>
- (snapshots-5, 2023-07-06 11:20) <br>
- (snapshots-6, 2023-07-06 11:30) <br>
- (snapshots-7, 2023-07-06 11:35) <br>
+ (snapshots-4, 2023-07-06 11:00) <br/>
+ (snapshots-5, 2023-07-06 11:20) <br/>
+ (snapshots-6, 2023-07-06 11:30) <br/>
+ (snapshots-7, 2023-07-06 11:35) <br/>
(snapshots-8, 2023-07-06 11:36)
</td>
<td>
@@ -218,7 +219,7 @@ Please note that too short retain time or too small retain number may result in:
expires, at which point the batch query will read a deleted snapshot.
- Streaming reading jobs on table files fail to restart.
When the job restarts, the snapshot it recorded may have expired. (You can use
- [Consumer Id]({{< ref "flink/sql-query#consumer-id" >}}) to protect streaming reading
+ [Consumer Id](../flink/consumer-id) to protect streaming reading
in a small retain time of snapshot expiration).
By default, paimon will delete expired snapshots synchronously. When there are too
@@ -230,9 +231,9 @@ may fail to complete successfully.
## Manually expire snapshot
Manually expire a table's snapshots
-{{< tabs "manually_expire_snapshot" >}}
+<Tabs groupId="manually-expire-snapshot">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following command:
@@ -246,15 +247,15 @@ CALL sys.expire_snapshots(`table` => 'database_name.table_name', older_than => '
CALL sys.expire_snapshots(`table` => 'database_name.table_name', older_than => '2024-01-01 12:00:00', max_deletes => 10, options => 'snapshot.expire.limit=1')
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action" >}}
+<TabItem value="flink-action" label="Flink Action">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
expire_snapshots \
--warehouse <warehouse-path> \
--identifier <identifier> \
@@ -266,9 +267,9 @@ Run the following command:
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark" >}}
+<TabItem value="spark" label="Spark">
Run the following sql:
@@ -276,17 +277,17 @@ Run the following sql:
CALL sys.expire_snapshots(table => 'database_name.table_name', retain_max => 10, options => 'snapshot.expire.limit=1');
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Rollback to Snapshot
Rollback a table to a specific snapshot ID.
-{{< tabs "rollback-to" >}}
+<Tabs groupId="rollback-to">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following command:
@@ -294,15 +295,15 @@ Run the following command:
CALL sys.rollback_to(`table` => 'database_name.table_name', snapshot_id => <snasphot-id>);
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action" >}}
+<TabItem value="flink-action" label="Flink Action">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
rollback_to \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -311,9 +312,9 @@ Run the following command:
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Java API" >}}
+<TabItem value="java-api" label="Java API">
```java
import org.apache.paimon.table.Table;
@@ -338,9 +339,9 @@ public class RollbackTo {
}
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark" >}}
+<TabItem value="spark" label="Spark">
Run the following sql:
@@ -348,9 +349,9 @@ Run the following sql:
CALL sys.rollback(table => 'database_name.table_name', snapshot => snasphot_id);
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Remove Orphan Files
@@ -358,21 +359,23 @@ Paimon files are deleted physically only when expiring snapshots. However, it is
when deleting files, so that there may exist files that are not used by Paimon snapshots (so-called "orphan files"). You can
submit a `remove_orphan_files` job to clean them:
-{{< tabs "remove_orphan_files" >}}
+<Tabs groupId="remove-orphan-files">
+
+<TabItem value="spark-sql-flink-sql" label="Spark SQL/Flink SQL">
-{{< tab "Spark SQL/Flink SQL" >}}
```sql
CALL sys.remove_orphan_files(`table` => 'my_db.my_table', [older_than => '2023-10-31 12:00:00'])
CALL sys.remove_orphan_files(`table` => 'my_db.*', [older_than => '2023-10-31 12:00:00'])
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
remove_orphan_files \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -387,7 +390,7 @@ To avoid deleting files that are newly added by other writing jobs, this action
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
remove_orphan_files \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -397,6 +400,6 @@ To avoid deleting files that are newly added by other writing jobs, this action
The table can be `*` to clean all tables in the database.
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
\ No newline at end of file
+</Tabs>
\ No newline at end of file
diff --git a/docs/content/maintenance/manage-tags.md b/docs/docs/maintenance/manage-tags.mdx
similarity index 84%
rename from docs/content/maintenance/manage-tags.md
rename to docs/docs/maintenance/manage-tags.mdx
index 2127c1bf23..78e588e5f5 100644
--- a/docs/content/maintenance/manage-tags.md
+++ b/docs/docs/maintenance/manage-tags.mdx
@@ -1,10 +1,11 @@
---
title: "Manage Tags"
-weight: 8
-type: docs
-aliases:
-- /maintenance/manage-tags.html
+sidebar_position: 8
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -44,10 +45,12 @@ You can set creation mode by table option `'tag.automatic-creation'`. Supported
- `watermark`: Create TAG based on the watermark of the Sink input.
- `batch`: In a batch processing scenario, a tag is generated after the current task is completed.
-{{< hint info >}}
+:::info
+
If you choose Watermark, you may need to specify the time zone of watermark, if watermark is not in the
UTC time zone, please configure `'sink.watermark-time-zone'`.
-{{< /hint >}}
+
+:::
**Step 2: Choose Creation Period**
@@ -88,15 +91,15 @@ SELECT * FROM t VERSION AS OF '2023-07-26';
SELECT * FROM paimon_incremental_query('t', '2023-07-25', '2023-07-26');
```
-See [Query Tables]({{< ref "spark/sql-query" >}}) to see more query for Spark.
+See [Query Tables](../spark/sql-query) to see more query for Spark.
## Create Tags
You can create a tag with given name and snapshot ID.
-{{< tabs "create-tag" >}}
+<Tabs groupId="create-tag">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following command:
@@ -106,15 +109,15 @@ CALL sys.create_tag(`table` => 'database_name.table_name', tag => 'tag_name', [s
If `snapshot_id` unset, snapshot_id defaults to the latest.
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action" >}}
+<TabItem value="flink-action" label="Flink Action">
- Run the following command:
+Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
create_tag \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -127,9 +130,9 @@ If `snapshot_id` unset, snapshot_id defaults to the latest.
If `snapshot` unset, snapshot_id defaults to the latest.
-{{< /tab >}}
+</TabItem>
-{{< tab "Java API" >}}
+<TabItem value="java-api" label="Java API">
```java
import org.apache.paimon.table.Table;
@@ -144,9 +147,10 @@ public class CreateTag {
}
```
-{{< /tab >}}
+</TabItem>
+
+<TabItem value="spark" label="Spark">
-{{< tab "Spark" >}}
Run the following sql:
```sql
CALL sys.create_tag(table => 'test.t', tag => 'test_tag', snapshot => 2);
@@ -162,17 +166,17 @@ To create a tag based on the latest snapshot id, run the following sql:
CALL sys.create_tag(table => 'test.t', tag => 'test_tag');
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Delete Tags
You can delete a tag by its name.
-{{< tabs "delete-tag" >}}
+<Tabs groupId="delete-tag">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following command:
@@ -180,15 +184,15 @@ Run the following command:
CALL sys.delete_tag(`table` => 'database_name.table_name', tag => 'tag_name');
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action" >}}
+<TabItem value="flink-action" label="Flink Action">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
delete_tag \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -197,9 +201,9 @@ Run the following command:
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Java API" >}}
+<TabItem value="java-api" label="Java API">
```java
import org.apache.paimon.table.Table;
@@ -213,27 +217,27 @@ public class DeleteTag {
}
```
-{{< /tab >}}
+</TabItem>
+<TabItem value="spark" label="Spark">
-{{< tab "Spark" >}}
Run the following sql:
```sql
CALL sys.delete_tag(table => 'test.t', tag => 'test_tag');
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
## Rollback to Tag
Rollback table to a specific tag. All snapshots and tags whose snapshot id is larger than the tag will be deleted (and
the data will be deleted too).
-{{< tabs "rollback-to" >}}
+<Tabs groupId="rollback-to">
-{{< tab "Flink SQL" >}}
+<TabItem value="flink-sql" label="Flink SQL">
Run the following command:
@@ -241,15 +245,15 @@ Run the following command:
CALL sys.rollback_to(`table` => 'database_name.table_name', tag => 'tag_name');
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Flink Action" >}}
+<TabItem value="flink-action" label="Flink Action">
Run the following command:
```bash
<FLINK_HOME>/bin/flink run \
- /path/to/paimon-flink-action-{{< version >}}.jar \
+ /path/to/paimon-flink-action-@@VERSION@@.jar \
rollback_to \
--warehouse <warehouse-path> \
--database <database-name> \
@@ -258,9 +262,9 @@ Run the following command:
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Java API" >}}
+<TabItem value="java-api" label="Java API">
```java
import org.apache.paimon.table.Table;
@@ -283,9 +287,9 @@ public class RollbackTo {
}
```
-{{< /tab >}}
+</TabItem>
-{{< tab "Spark" >}}
+<TabItem value="spark" label="Spark">
Run the following sql:
@@ -293,6 +297,6 @@ Run the following sql:
CALL sys.rollback(table => 'test.t', version => '2');
```
-{{< /tab >}}
+</TabItem>
-{{< /tabs >}}
+</Tabs>
diff --git a/docs/content/maintenance/metrics.md b/docs/docs/maintenance/metrics.md
similarity index 98%
rename from docs/content/maintenance/metrics.md
rename to docs/docs/maintenance/metrics.md
index 747a3179d7..f6e1576dea 100644
--- a/docs/content/maintenance/metrics.md
+++ b/docs/docs/maintenance/metrics.md
@@ -1,10 +1,8 @@
---
title: "Metrics"
-weight: 9
-type: docs
-aliases:
-- /maintenance/metrics.html
+sidebar_position: 9
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -321,7 +319,7 @@ Below is lists of Paimon built-in metrics. They are summarized into types of sca
Paimon has implemented bridging metrics to Flink's metrics system, which can be reported by Flink, and the lifecycle of metric groups are managed by Flink.
-Please join the `<scope>.<infix>.<metric_name>` to get the complete metric identifier when using Flink to access Paimon, `metric_name` can be got from [Metric List]({{< ref "maintenance/metrics#metrics-list" >}}).
+Please join the `<scope>.<infix>.<metric_name>` to get the complete metric identifier when using Flink to access Paimon, `metric_name` can be got from [Metric List](./metrics#metrics-list).
For example, the identifier of metric `lastPartitionsWritten` for table `word_count` in Flink job named `insert_word_count` is:
@@ -331,10 +329,12 @@ From Flink Web-UI, go to the committer operator's metrics, it's shown as:
`0.Global_Committer___word_count.paimon.table.word_count.commit.lastPartitionsWritten`.
-{{< hint info >}}
+:::info
+
1. Please refer to [System Scope](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#system-scope) to understand Flink `scope`
2. Scan metrics are only supported by Flink versions >= 1.18
-{{< /hint >}}
+
+:::
<table class="table table-bordered">
<thead>
@@ -420,9 +420,11 @@ When using Flink to read and write, Paimon has implemented some key standard Fli
</tbody>
</table>
-{{< hint info >}}
+:::info
+
Please note that if you specified `consumer-id` in your streaming query, the level of source metrics should turn into the reader operator, which is behind the `Monitor` operator.
-{{< /hint >}}
+
+:::
#### Sink Metrics (Flink)
diff --git a/docs/content/maintenance/rescale-bucket.md b/docs/docs/maintenance/rescale-bucket.md
similarity index 98%
rename from docs/content/maintenance/rescale-bucket.md
rename to docs/docs/maintenance/rescale-bucket.md
index 22b134c2f3..1a30452534 100644
--- a/docs/content/maintenance/rescale-bucket.md
+++ b/docs/docs/maintenance/rescale-bucket.md
@@ -1,10 +1,8 @@
---
title: "Rescale Bucket"
-weight: 7
-type: docs
-aliases:
-- /maintenance/rescale-bucket.html
+sidebar_position: 7
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/maintenance/write-performance.md b/docs/docs/maintenance/write-performance.md
similarity index 97%
rename from docs/content/maintenance/write-performance.md
rename to docs/docs/maintenance/write-performance.md
index b224c9dc44..670260715e 100644
--- a/docs/content/maintenance/write-performance.md
+++ b/docs/docs/maintenance/write-performance.md
@@ -1,10 +1,8 @@
---
title: "Write Performance"
-weight: 3
-type: docs
-aliases:
-- /maintenance/write-performance.html
+sidebar_position: 3
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -40,7 +38,7 @@ large impact on write performance, if it is a snapshot / full synchronization ph
then enable them again in the incremental phase.
If you find that the input of the job shows a jagged pattern in the case of backpressure, it may be imbalanced work
-nodes. You can consider turning on [Asynchronous Compaction]({{< ref "primary-key-table/compaction#asynchronous-compaction" >}}) to observe if the
+nodes. You can consider turning on [Asynchronous Compaction](../primary-key-table/compaction#asynchronous-compaction) to observe if the
throughput is increased.
## Parallelism
diff --git a/docs/content/migration/clone-to-paimon.md b/docs/docs/migration/clone-to-paimon.md
similarity index 91%
rename from docs/content/migration/clone-to-paimon.md
rename to docs/docs/migration/clone-to-paimon.md
index a6665deb20..b8383875f3 100644
--- a/docs/content/migration/clone-to-paimon.md
+++ b/docs/docs/migration/clone-to-paimon.md
@@ -1,10 +1,8 @@
---
title: "Clone To Paimon"
-weight: 5
-type: docs
-aliases:
-- /migration/clone.html
+sidebar_position: 5
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -38,7 +36,7 @@ target table will be append table.
## Clone Hive Table
```bash
-<FLINK_HOME>/flink run ./paimon-flink-action-{{< version >}}.jar \
+<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \
clone \
--database default \
--table hivetable \
@@ -56,7 +54,7 @@ You can use filter spec to specify the filtering condition for the partition.
## Clone Hive Database
```bash
-<FLINK_HOME>/flink run ./paimon-flink-action-{{< version >}}.jar \
+<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \
clone \
--database default \
--catalog_conf metastore=hive \
@@ -74,7 +72,7 @@ The format is `<database1>.<table1>,<database2>.<table2>,<database3>.<table3>`.
## Clone Hive Catalog
```bash
-<FLINK_HOME>/flink run ./paimon-flink-action-{{< version >}}.jar \
+<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \
clone \
--catalog_conf metastore=hive \
--catalog_conf uri=thrift://localhost:9088 \
diff --git a/docs/content/concepts/rest/_index.md b/docs/docs/migration/index.md
similarity index 93%
copy from docs/content/concepts/rest/_index.md
copy to docs/docs/migration/index.md
index eacaa231ca..bc01176fbc 100644
--- a/docs/content/concepts/rest/_index.md
+++ b/docs/docs/migration/index.md
@@ -1,8 +1,8 @@
---
-title: RESTCatalog
-bookCollapseSection: true
-weight: 5
+title: "Migration"
+sidebar_position: 97
---
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
diff --git a/docs/content/migration/migration-from-hive.md b/docs/docs/migration/migration-from-hive.mdx
similarity index 80%
rename from docs/content/migration/migration-from-hive.md
rename to docs/docs/migration/migration-from-hive.mdx
index 03ff208ee5..2cce328521 100644
--- a/docs/content/migration/migration-from-hive.md
+++ b/docs/docs/migration/migration-from-hive.mdx
@@ -1,10 +1,11 @@
---
title: "Migration From Hive"
-weight: 1
-type: docs
-aliases:
-- /migration/migration-from-hive.html
+sidebar_position: 1
---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -28,7 +29,7 @@ under the License.
Apache Hive supports ORC, Parquet file formats that could be migrated to Paimon.
When migrating data to a paimon table, the origin table will be permanently disappeared. So please back up your data if you
-still need the original table. The migrated table will be [append table]({{< ref "append-table/overview" >}}).
+still need the original table. The migrated table will be [append table](../append-table/overview).
Now, we can use paimon hive catalog with Migrate Table Procedure to totally migrate a table from hive to paimon.
At the same time, you can use paimon hive catalog with Migrate Database Procedure to fully synchronize all tables in the database to paimon.
@@ -38,12 +39,14 @@ At the same time, you can use paimon hive catalog with Migrate Database Procedur
These three actions now support file format of hive "orc" and "parquet" and "avro".
-<span style="color: red; "> **We highly recommend to back up hive table data before migrating, because migrating action is not atomic. If been interrupted while migrating, you may lose your data.** </span>
+<span style={{color: "red"}}> **We highly recommend to back up hive table data before migrating, because migrating action is not atomic. If been interrupted while migrating, you may lose your data.** </span>
## Migrate Hive Table
-{{< tabs "migrate table" >}}
-{{< tab "Flink SQL" >}}
+<Tabs groupId="migrate-table">
+
+<TabItem value="flink-sql" label="Flink SQL">
+
```sql
CREATE CATALOG PAIMON WITH (
'type'='paimon',
@@ -63,10 +66,13 @@ CALL sys.migrate_table(
-- delete_origin => false,
options => 'file.format=orc');
```
-{{< /tab >}}
-{{< tab "Flink Action" >}}
+
+</TabItem>
+
+<TabItem value="flink-action" label="Flink Action">
+
```bash
-<FLINK_HOME>/flink run ./paimon-flink-action-{{< version >}}.jar \
+<FLINK_HOME>/flink run ./paimon-flink-action-@@VERSION@@.jar \
migrate_table \
--warehouse /path/to/warehouse \
--catalog_conf uri=thrift://localhost:9083 \
@@ -74,15 +80,19 @@ migrate_table \
--source_type hive \
--table default.hive_or_paimon
```
-{{< /tab >}}
-{{< /tabs >}}
+
+</TabItem>
+
+</Tabs>
After invoke, "hivetable" will totally convert to paimon format. Writing and reading the table by old "hive way" will fail.
## Migrate Hive Database
... 19250 lines suppressed ...