You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by xu...@apache.org on 2025/12/17 05:21:22 UTC
(hudi) branch asf-site updated: chore(site): fix docs link generation and config docs gen (#17619)
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 234c0389bcdb chore(site): fix docs link generation and config docs gen (#17619)
234c0389bcdb is described below
commit 234c0389bcdb354218b9e1d5ad6969a525082f1b
Author: Shiyan Xu <27...@users.noreply.github.com>
AuthorDate: Tue Dec 16 23:21:15 2025 -0600
chore(site): fix docs link generation and config docs gen (#17619)
---
.github/workflows/asf-site.ci.yml | 85 ++++++++++++++++------
README.md | 85 +++++++++++++++-------
.../hudi/utils/HoodieConfigDocGenerator.java | 4 +-
.../version-0.14.0/basic_configurations.md | 4 +-
website/versioned_docs/version-0.14.0/cli.md | 2 +-
website/versioned_docs/version-0.14.0/cloud.md | 18 ++---
.../versioned_docs/version-0.14.0/clustering.md | 4 +-
.../versioned_docs/version-0.14.0/compaction.md | 6 +-
.../version-0.14.0/configurations.md | 2 +-
.../versioned_docs/version-0.14.0/deployment.md | 10 +--
website/versioned_docs/version-0.14.0/faq.md | 2 +-
.../versioned_docs/version-0.14.0/file_layouts.md | 4 +-
.../versioned_docs/version-0.14.0/file_sizing.md | 2 +-
.../version-0.14.0/flink-quick-start-guide.md | 36 ++++-----
.../version-0.14.0/hoodie_cleaner.md | 2 +-
.../version-0.14.0/hoodie_streaming_ingestion.md | 8 +-
website/versioned_docs/version-0.14.0/indexing.md | 4 +-
website/versioned_docs/version-0.14.0/metadata.md | 12 +--
.../version-0.14.0/metadata_indexing.md | 6 +-
website/versioned_docs/version-0.14.0/metrics.md | 10 +--
.../version-0.14.0/migration_guide.md | 4 +-
website/versioned_docs/version-0.14.0/overview.mdx | 26 +++----
.../versioned_docs/version-0.14.0/performance.md | 12 +--
.../version-0.14.0/precommit_validator.md | 2 +-
.../versioned_docs/version-0.14.0/procedures.md | 42 +++++------
.../versioned_docs/version-0.14.0/querying_data.md | 10 +--
.../version-0.14.0/quick-start-guide.md | 32 ++++----
.../version-0.14.0/record_payload.md | 4 +-
website/versioned_docs/version-0.14.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.14.0/sql_ddl.md | 8 +-
website/versioned_docs/version-0.14.0/sql_dml.md | 8 +-
.../versioned_docs/version-0.14.0/sql_queries.md | 8 +-
website/versioned_docs/version-0.14.0/structure.md | 2 +-
.../syncing_aws_glue_data_catalog.md | 2 +-
.../version-0.14.0/syncing_metastore.md | 2 +-
website/versioned_docs/version-0.14.0/use_cases.md | 12 +--
.../version-0.14.0/write_operations.md | 20 ++---
.../versioned_docs/version-0.14.0/writing_data.md | 18 ++---
.../version-0.14.1/basic_configurations.md | 4 +-
website/versioned_docs/version-0.14.1/cli.md | 2 +-
website/versioned_docs/version-0.14.1/cloud.md | 18 ++---
.../versioned_docs/version-0.14.1/clustering.md | 4 +-
.../versioned_docs/version-0.14.1/compaction.md | 6 +-
.../version-0.14.1/concurrency_control.md | 2 +-
.../versioned_docs/version-0.14.1/deployment.md | 10 +--
website/versioned_docs/version-0.14.1/faq.md | 14 ++--
.../version-0.14.1/faq_design_and_concepts.md | 10 +--
.../versioned_docs/version-0.14.1/faq_general.md | 10 +--
.../version-0.14.1/faq_querying_tables.md | 2 +-
.../versioned_docs/version-0.14.1/faq_storage.md | 12 +--
.../version-0.14.1/faq_table_services.md | 6 +-
.../version-0.14.1/faq_writing_tables.md | 34 ++++-----
.../versioned_docs/version-0.14.1/file_layouts.md | 4 +-
.../versioned_docs/version-0.14.1/file_sizing.md | 2 +-
.../version-0.14.1/flink-quick-start-guide.md | 32 ++++----
.../version-0.14.1/hoodie_cleaner.md | 2 +-
.../version-0.14.1/hoodie_streaming_ingestion.md | 8 +-
website/versioned_docs/version-0.14.1/indexing.md | 4 +-
website/versioned_docs/version-0.14.1/metadata.md | 12 +--
.../version-0.14.1/metadata_indexing.md | 6 +-
website/versioned_docs/version-0.14.1/metrics.md | 10 +--
.../version-0.14.1/migration_guide.md | 4 +-
website/versioned_docs/version-0.14.1/overview.mdx | 26 +++----
.../versioned_docs/version-0.14.1/performance.md | 12 +--
.../version-0.14.1/precommit_validator.md | 2 +-
.../versioned_docs/version-0.14.1/procedures.md | 42 +++++------
.../versioned_docs/version-0.14.1/querying_data.md | 10 +--
.../version-0.14.1/quick-start-guide.md | 32 ++++----
.../version-0.14.1/record_payload.md | 4 +-
website/versioned_docs/version-0.14.1/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.14.1/sql_ddl.md | 10 +--
website/versioned_docs/version-0.14.1/sql_dml.md | 6 +-
.../versioned_docs/version-0.14.1/sql_queries.md | 8 +-
website/versioned_docs/version-0.14.1/structure.md | 2 +-
.../syncing_aws_glue_data_catalog.md | 2 +-
.../version-0.14.1/syncing_metastore.md | 2 +-
website/versioned_docs/version-0.14.1/use_cases.md | 12 +--
.../version-0.14.1/write_operations.md | 20 ++---
.../versioned_docs/version-0.14.1/writing_data.md | 30 ++++----
.../version-0.15.0/basic_configurations.md | 4 +-
website/versioned_docs/version-0.15.0/cli.md | 2 +-
website/versioned_docs/version-0.15.0/cloud.md | 18 ++---
.../versioned_docs/version-0.15.0/clustering.md | 4 +-
.../versioned_docs/version-0.15.0/compaction.md | 6 +-
.../version-0.15.0/concurrency_control.md | 2 +-
.../versioned_docs/version-0.15.0/deployment.md | 10 +--
website/versioned_docs/version-0.15.0/faq.md | 14 ++--
.../version-0.15.0/faq_design_and_concepts.md | 10 +--
.../versioned_docs/version-0.15.0/faq_general.md | 10 +--
.../version-0.15.0/faq_reading_tables.md | 2 +-
.../versioned_docs/version-0.15.0/faq_storage.md | 12 +--
.../version-0.15.0/faq_table_services.md | 8 +-
.../version-0.15.0/faq_writing_tables.md | 36 ++++-----
.../versioned_docs/version-0.15.0/file_layouts.md | 4 +-
.../versioned_docs/version-0.15.0/file_sizing.md | 2 +-
.../version-0.15.0/flink-quick-start-guide.md | 30 ++++----
.../version-0.15.0/hoodie_cleaner.md | 2 +-
.../version-0.15.0/hoodie_streaming_ingestion.md | 4 +-
.../versioned_docs/version-0.15.0/hudi_stack.md | 18 ++---
website/versioned_docs/version-0.15.0/indexing.md | 4 +-
.../version-0.15.0/ingestion_flink.md | 4 +-
website/versioned_docs/version-0.15.0/metadata.md | 12 +--
.../version-0.15.0/metadata_indexing.md | 6 +-
website/versioned_docs/version-0.15.0/metrics.md | 10 +--
.../version-0.15.0/migration_guide.md | 4 +-
website/versioned_docs/version-0.15.0/overview.mdx | 28 +++----
.../versioned_docs/version-0.15.0/performance.md | 12 +--
.../version-0.15.0/precommit_validator.md | 4 +-
.../versioned_docs/version-0.15.0/procedures.md | 42 +++++------
.../python-rust-quick-start-guide.md | 2 +-
.../versioned_docs/version-0.15.0/querying_data.md | 10 +--
.../version-0.15.0/quick-start-guide.md | 32 ++++----
.../version-0.15.0/record_payload.md | 4 +-
website/versioned_docs/version-0.15.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-0.15.0/sql_ddl.md | 10 +--
website/versioned_docs/version-0.15.0/sql_dml.md | 8 +-
.../versioned_docs/version-0.15.0/sql_queries.md | 8 +-
website/versioned_docs/version-0.15.0/structure.md | 2 +-
.../syncing_aws_glue_data_catalog.md | 2 +-
.../version-0.15.0/syncing_metastore.md | 2 +-
website/versioned_docs/version-0.15.0/use_cases.md | 12 +--
.../version-0.15.0/write_operations.md | 18 ++---
.../versioned_docs/version-0.15.0/writing_data.md | 18 ++---
.../version-1.0.0/basic_configurations.md | 4 +-
website/versioned_docs/version-1.0.0/cleaning.md | 2 +-
website/versioned_docs/version-1.0.0/cli.md | 2 +-
website/versioned_docs/version-1.0.0/cloud.md | 18 ++---
website/versioned_docs/version-1.0.0/clustering.md | 4 +-
website/versioned_docs/version-1.0.0/compaction.md | 6 +-
.../version-1.0.0/concurrency_control.md | 6 +-
website/versioned_docs/version-1.0.0/deployment.md | 10 +--
website/versioned_docs/version-1.0.0/faq.md | 14 ++--
.../version-1.0.0/faq_design_and_concepts.md | 10 +--
.../versioned_docs/version-1.0.0/faq_general.md | 10 +--
.../version-1.0.0/faq_reading_tables.md | 2 +-
.../versioned_docs/version-1.0.0/faq_storage.md | 12 +--
.../version-1.0.0/faq_table_services.md | 8 +-
.../version-1.0.0/faq_writing_tables.md | 36 ++++-----
.../versioned_docs/version-1.0.0/file_sizing.md | 2 +-
.../version-1.0.0/flink-quick-start-guide.md | 30 ++++----
.../version-1.0.0/hoodie_streaming_ingestion.md | 6 +-
website/versioned_docs/version-1.0.0/hudi_stack.md | 24 +++---
website/versioned_docs/version-1.0.0/indexes.md | 12 +--
.../version-1.0.0/ingestion_flink.md | 4 +-
website/versioned_docs/version-1.0.0/metadata.md | 8 +-
.../version-1.0.0/metadata_indexing.md | 10 +--
website/versioned_docs/version-1.0.0/metrics.md | 10 +--
.../version-1.0.0/migration_guide.md | 4 +-
website/versioned_docs/version-1.0.0/overview.mdx | 28 +++----
.../versioned_docs/version-1.0.0/performance.md | 12 +--
.../version-1.0.0/precommit_validator.md | 4 +-
website/versioned_docs/version-1.0.0/procedures.md | 42 +++++------
.../version-1.0.0/python-rust-quick-start-guide.md | 2 +-
.../versioned_docs/version-1.0.0/querying_data.md | 10 +--
.../version-1.0.0/quick-start-guide.md | 38 +++++-----
.../versioned_docs/version-1.0.0/record_merger.md | 8 +-
website/versioned_docs/version-1.0.0/s3_hoodie.md | 2 +-
website/versioned_docs/version-1.0.0/sql_ddl.md | 22 +++---
website/versioned_docs/version-1.0.0/sql_dml.md | 8 +-
.../versioned_docs/version-1.0.0/sql_queries.md | 12 +--
.../version-1.0.0/storage_layouts.md | 4 +-
website/versioned_docs/version-1.0.0/structure.md | 2 +-
.../version-1.0.0/syncing_aws_glue_data_catalog.md | 2 +-
.../version-1.0.0/syncing_metastore.md | 2 +-
website/versioned_docs/version-1.0.0/use_cases.md | 10 +--
.../version-1.0.0/write_operations.md | 18 ++---
.../versioned_docs/version-1.0.0/writing_data.md | 18 ++---
.../version-1.0.1/basic_configurations.md | 4 +-
website/versioned_docs/version-1.0.1/cleaning.md | 2 +-
website/versioned_docs/version-1.0.1/cli.md | 2 +-
website/versioned_docs/version-1.0.1/cloud.md | 18 ++---
website/versioned_docs/version-1.0.1/clustering.md | 4 +-
website/versioned_docs/version-1.0.1/compaction.md | 6 +-
.../version-1.0.1/concurrency_control.md | 4 +-
website/versioned_docs/version-1.0.1/deployment.md | 10 +--
website/versioned_docs/version-1.0.1/faq.md | 14 ++--
.../version-1.0.1/faq_design_and_concepts.md | 10 +--
.../versioned_docs/version-1.0.1/faq_general.md | 10 +--
.../version-1.0.1/faq_reading_tables.md | 2 +-
.../versioned_docs/version-1.0.1/faq_storage.md | 12 +--
.../version-1.0.1/faq_table_services.md | 8 +-
.../version-1.0.1/faq_writing_tables.md | 36 ++++-----
.../versioned_docs/version-1.0.1/file_sizing.md | 2 +-
.../version-1.0.1/flink-quick-start-guide.md | 30 ++++----
.../version-1.0.1/hoodie_streaming_ingestion.md | 6 +-
website/versioned_docs/version-1.0.1/hudi_stack.md | 24 +++---
website/versioned_docs/version-1.0.1/indexes.md | 12 +--
.../version-1.0.1/ingestion_flink.md | 4 +-
website/versioned_docs/version-1.0.1/metadata.md | 8 +-
.../version-1.0.1/metadata_indexing.md | 10 +--
website/versioned_docs/version-1.0.1/metrics.md | 10 +--
.../version-1.0.1/migration_guide.md | 4 +-
website/versioned_docs/version-1.0.1/overview.mdx | 28 +++----
.../versioned_docs/version-1.0.1/performance.md | 12 +--
.../version-1.0.1/precommit_validator.md | 4 +-
website/versioned_docs/version-1.0.1/procedures.md | 42 +++++------
.../version-1.0.1/python-rust-quick-start-guide.md | 2 +-
.../versioned_docs/version-1.0.1/querying_data.md | 10 +--
.../version-1.0.1/quick-start-guide.md | 38 +++++-----
.../versioned_docs/version-1.0.1/record_merger.md | 8 +-
website/versioned_docs/version-1.0.1/s3_hoodie.md | 2 +-
website/versioned_docs/version-1.0.1/sql_ddl.md | 22 +++---
website/versioned_docs/version-1.0.1/sql_dml.md | 8 +-
.../versioned_docs/version-1.0.1/sql_queries.md | 12 +--
.../version-1.0.1/storage_layouts.md | 4 +-
website/versioned_docs/version-1.0.1/structure.md | 2 +-
.../version-1.0.1/syncing_aws_glue_data_catalog.md | 2 +-
.../version-1.0.1/syncing_metastore.md | 2 +-
website/versioned_docs/version-1.0.1/use_cases.md | 10 +--
.../version-1.0.1/write_operations.md | 18 ++---
.../versioned_docs/version-1.0.1/writing_data.md | 18 ++---
.../version-1.0.2/basic_configurations.md | 4 +-
website/versioned_docs/version-1.0.2/cleaning.md | 2 +-
website/versioned_docs/version-1.0.2/cli.md | 2 +-
website/versioned_docs/version-1.0.2/cloud.md | 18 ++---
website/versioned_docs/version-1.0.2/clustering.md | 4 +-
website/versioned_docs/version-1.0.2/compaction.md | 6 +-
.../version-1.0.2/concurrency_control.md | 4 +-
website/versioned_docs/version-1.0.2/deployment.md | 10 +--
website/versioned_docs/version-1.0.2/faq.md | 14 ++--
.../version-1.0.2/faq_design_and_concepts.md | 10 +--
.../versioned_docs/version-1.0.2/faq_general.md | 10 +--
.../version-1.0.2/faq_reading_tables.md | 2 +-
.../versioned_docs/version-1.0.2/faq_storage.md | 12 +--
.../version-1.0.2/faq_table_services.md | 8 +-
.../version-1.0.2/faq_writing_tables.md | 36 ++++-----
.../versioned_docs/version-1.0.2/file_sizing.md | 2 +-
.../version-1.0.2/flink-quick-start-guide.md | 30 ++++----
.../version-1.0.2/hoodie_streaming_ingestion.md | 6 +-
website/versioned_docs/version-1.0.2/hudi_stack.md | 24 +++---
website/versioned_docs/version-1.0.2/indexes.md | 12 +--
.../version-1.0.2/ingestion_flink.md | 4 +-
website/versioned_docs/version-1.0.2/metadata.md | 8 +-
.../version-1.0.2/metadata_indexing.md | 10 +--
website/versioned_docs/version-1.0.2/metrics.md | 10 +--
.../version-1.0.2/migration_guide.md | 4 +-
website/versioned_docs/version-1.0.2/overview.mdx | 28 +++----
.../versioned_docs/version-1.0.2/performance.md | 12 +--
.../version-1.0.2/precommit_validator.md | 4 +-
website/versioned_docs/version-1.0.2/procedures.md | 42 +++++------
.../version-1.0.2/python-rust-quick-start-guide.md | 2 +-
.../versioned_docs/version-1.0.2/querying_data.md | 10 +--
.../version-1.0.2/quick-start-guide.md | 38 +++++-----
.../versioned_docs/version-1.0.2/record_merger.md | 8 +-
website/versioned_docs/version-1.0.2/s3_hoodie.md | 2 +-
website/versioned_docs/version-1.0.2/sql_ddl.md | 22 +++---
website/versioned_docs/version-1.0.2/sql_dml.md | 8 +-
.../versioned_docs/version-1.0.2/sql_queries.md | 12 +--
.../version-1.0.2/storage_layouts.md | 4 +-
website/versioned_docs/version-1.0.2/structure.md | 2 +-
.../version-1.0.2/syncing_aws_glue_data_catalog.md | 2 +-
.../version-1.0.2/syncing_metastore.md | 2 +-
website/versioned_docs/version-1.0.2/use_cases.md | 10 +--
.../version-1.0.2/write_operations.md | 18 ++---
.../versioned_docs/version-1.0.2/writing_data.md | 18 ++---
255 files changed, 1487 insertions(+), 1417 deletions(-)
diff --git a/.github/workflows/asf-site.ci.yml b/.github/workflows/asf-site.ci.yml
index a08cc4847c78..350d03f03f56 100644
--- a/.github/workflows/asf-site.ci.yml
+++ b/.github/workflows/asf-site.ci.yml
@@ -41,10 +41,10 @@ jobs:
else
echo "β
No files under content/ directory."
fi
- - name: Check for absolute /docs/ paths in markdown links
+ - name: Check docs markdown link format
if: github.event_name == 'pull_request'
run: |
- echo "π Checking for absolute /docs/ paths in markdown links..."
+ echo "π Checking docs markdown link format..."
# Get the base branch
git fetch origin ${{ github.base_ref }}
@@ -52,11 +52,11 @@ jobs:
# Get list of changed files in the PR
CHANGED_FILES=$(git diff --name-only origin/${{ github.base_ref }} HEAD)
- # Filter for .md and .mdx files in website/docs/
- DOCS_FILES=$(echo "$CHANGED_FILES" | grep -E '^website/docs/.*\.(md|mdx)$' || true)
+ # Filter for .md and .mdx files in website/docs/ and website/versioned_docs/
+ DOCS_FILES=$(echo "$CHANGED_FILES" | grep -E '^website/(docs|versioned_docs)/.*\.(md|mdx)$' || true)
if [ -z "$DOCS_FILES" ]; then
- echo "β
No markdown files changed in website/docs/"
+ echo "β
No markdown files changed in docs directories"
exit 0
fi
@@ -64,18 +64,19 @@ jobs:
echo "$DOCS_FILES"
echo ""
- # Variable to track if we found any issues
- FOUND_ISSUES=false
- ISSUE_FILES=""
+ # Variables to track issues
+ FOUND_ABSOLUTE_PATH_ISSUES=false
+ FOUND_INVALID_RELATIVE_LINK_ISSUES=false
+ ABSOLUTE_PATH_FILES=""
+ INVALID_RELATIVE_LINK_FILES=""
- # Check each file for markdown links starting with /docs/
while IFS= read -r file; do
if [ -f "$file" ]; then
- # Look for markdown links with /docs/ paths
+ # Check 1: Look for absolute /docs/ paths
# Pattern: [text](/docs/...)
if grep -Hn '\[.*\](\s*/docs/[^)]*)' "$file" > /dev/null 2>&1; then
- FOUND_ISSUES=true
- ISSUE_FILES="${ISSUE_FILES}${file}\n"
+ FOUND_ABSOLUTE_PATH_ISSUES=true
+ ABSOLUTE_PATH_FILES="${ABSOLUTE_PATH_FILES}${file}\n"
echo "β Found absolute /docs/ paths in: $file"
echo " Lines with issues:"
grep -Hn '\[.*\](\s*/docs/[^)]*)' "$file" | while IFS= read -r line; do
@@ -83,29 +84,69 @@ jobs:
done
echo ""
fi
+
+ # Check 2: Look for invalid relative doc links
+ # - Missing .md extension: ](word#anchor) or ](word) where word starts with lowercase letter
+ # - Using ../ or ./ prefixes: ](../ or ](./
+ MISSING_MD=$(grep -En '\]\([a-z][a-z0-9_-]*(#|\))' "$file" 2>/dev/null | grep -v '\.md[#)]' || true)
+ RELATIVE_PREFIX=$(grep -En '\]\(\.\.?/' "$file" 2>/dev/null || true)
+ if [ -n "$MISSING_MD" ] || [ -n "$RELATIVE_PREFIX" ]; then
+ FOUND_INVALID_RELATIVE_LINK_ISSUES=true
+ INVALID_RELATIVE_LINK_FILES="${INVALID_RELATIVE_LINK_FILES}${file}\n"
+ echo "β Found invalid relative doc links in: $file"
+ echo " Lines with issues:"
+ { echo "$MISSING_MD"; echo "$RELATIVE_PREFIX"; } | grep -v '^$' | sort -t: -k1,1n -u | while IFS= read -r line; do
+ echo " $line"
+ done
+ echo ""
+ fi
fi
done <<< "$DOCS_FILES"
- if [ "$FOUND_ISSUES" = true ]; then
+ # Report results
+ HAS_ERRORS=false
+
+ if [ "$FOUND_ABSOLUTE_PATH_ISSUES" = true ]; then
+ HAS_ERRORS=true
echo ""
echo "========================================="
echo "β ERROR: Found absolute /docs/ paths in markdown links!"
echo "========================================="
echo ""
- echo "π Files with issues:"
- echo -e "$ISSUE_FILES"
- echo "π‘ SUGGESTION: Use relative paths instead of absolute paths."
+ echo "π Files with absolute path issues:"
+ echo -e "$ABSOLUTE_PATH_FILES"
+ echo "π‘ Use relative paths with .md extension instead."
+ echo " β Bad: [link text](/docs/guide/overview)"
+ echo " β
Good: [link text](overview.md)"
+ echo ""
+ fi
+
+ if [ "$FOUND_INVALID_RELATIVE_LINK_ISSUES" = true ]; then
+ HAS_ERRORS=true
+ echo ""
+ echo "========================================="
+ echo "β ERROR: Found invalid relative doc links!"
+ echo "========================================="
+ echo ""
+ echo "π Files with invalid relative links:"
+ echo -e "$INVALID_RELATIVE_LINK_FILES"
+ echo "π‘ Use just the filename with .md extension."
+ echo " β Bad: [link text](configurations#anchor) (missing .md)"
+ echo " β Bad: [link text](../configurations.md) (using ../)"
+ echo " β Bad: [link text](./configurations.md) (using ./)"
+ echo " β
Good: [link text](configurations.md#anchor)"
echo ""
- echo "Examples:"
- echo " β Bad: [link text](/docs/guide/overview)"
- echo " β
Good: [link text](overview) or [link text](../guide/overview)"
+ fi
+
+ if [ "$HAS_ERRORS" = true ]; then
echo ""
- echo "Relative paths ensure links work correctly across different versions"
- echo "and environments. Please update the links to use relative paths."
+ echo "The .md extension ensures Docusaurus resolves links at build time,"
+ echo "making them work correctly with trailing slashes and versioned docs."
+ echo "See README.md 'Linking docs' section for more details."
exit 1
else
echo ""
- echo "β
All markdown links use appropriate paths (no absolute /docs/ paths found)"
+ echo "β
All docs markdown links are properly formatted"
fi
- name: Prepare branch
run: |
diff --git a/README.md b/README.md
index 0b4c6c007297..e6ad949eff65 100644
--- a/README.md
+++ b/README.md
@@ -131,43 +131,72 @@ versioned_sidebars/version-0.7.0-sidebars.json
### Linking docs
-Relative paths work well. - Files will be linked to correct corresponding version.
+**IMPORTANT: Always use just the filename with `.md` extension when linking to other docs.**
-- Referring to versioned docs pages within `docs/` MUST USE relative paths.
- - **Good Example of linking.**
- Say we are updating a 0.12.0 version doc which is older.
+Docusaurus resolves file paths with `.md` extension at build time, converting them to absolute URLs. This ensures:
- ```md
- A [callback notification](writing_data#commit-notifications) is exposed
- ```
+- Links work correctly regardless of trailing slashes in URLs
+- Links automatically resolve to the correct version in versioned docs
- This automatically resolves to /docs/0.12.0/writing_data#commit-notifications.
- - **Bad example of linking.**
- Say we are updating a 0.12.0 version doc which is older.
+**DO NOT use `../` or `./` prefixes** - these break versioned docs behavior.
- ```md
- A [callback notification](/docs/writing_data#commit-notifications) is exposed
- ```
+#### Versioned docs linking
- This will resolve to the most recent release, specifically /docs/writing_data#commit-notifications . We do not want a 0.12.0 doc page to point to a page from a later release.
-- DO NOT use next version when linking.
- - Good Example of linking when you are working on unreleased version (from next version).
+Referring to versioned docs pages within `docs/` MUST USE relative paths with `.md` extension.
- ```md
- Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
- file slices and [cleaning](cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
- ```
+**Good Example of linking:**
- This automatically resolves to /docs/next/compaction and /docs/next/cleaning pages.
- - Bad Example of linking when you are working on unreleased version (from next version).
+Say we are updating a 0.12.0 version doc which is older.
- ```md
- Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/next/compaction) action merges logs and base files to produce new
- file slices and [cleaning](/docs/next/cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
- ```
+```md
+A [callback notification](writing_data.md#commit-notifications) is exposed
+```
+
+This automatically resolves to /docs/0.12.0/writing_data#commit-notifications.
+
+**Bad examples of linking:**
+
+```md
+A [callback notification](/docs/writing_data#commit-notifications) is exposed
+```
+
+This will resolve to the most recent release, specifically /docs/writing_data#commit-notifications. We do not want a 0.12.0 doc page to point to a page from a later release.
+
+```md
+A [callback notification](writing_data#commit-notifications) is exposed
+```
+
+Without `.md` extension, this becomes a relative URL that the browser resolves at runtime, which can break when the page URL has a trailing slash.
+
+```md
+A [callback notification](../writing_data.md#commit-notifications) is exposed
+```
+
+Using `../` or `./` prefixes breaks versioned docs behavior and should be avoided.
+
+#### DO NOT use next version when linking
+
+**Good Example** - linking when you are working on unreleased version (from next version):
+
+```md
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+file slices and [cleaning](cleaning.md) action gets rid of unused/older file slices to reclaim space on the file system.
+```
+
+This automatically resolves to /docs/next/compaction and /docs/next/cleaning pages.
+
+**Bad Example** - linking when you are working on unreleased version (from next version):
+
+```md
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/next/compaction) action merges logs and base files to produce new
+file slices and [cleaning](/docs/next/cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+```
+
+Even though it directly points to /docs/next which is intended target, this accumulates as tech debt when this copy of docs gets released, we will have an older doc always pointing to /docs/next/.
+
+#### Linking from outside docs
- Even though it directly points to /docs/next which is intended target, this accumulates as tech debt when this copy of docs gets released, we will hav a older doc always pointing to /docs/next/
-- Refer to `/docs/...` or `/docs/<version>/...` (when a specific version is needed) from pages that are outside of `/docs/` dir.
+Refer to `/docs/...` or `/docs/<version>/...` (when a specific version is needed) from pages that are outside of `/docs/` dir.
## Versions
diff --git a/hudi-utils/src/main/java/org/apache/hudi/utils/HoodieConfigDocGenerator.java b/hudi-utils/src/main/java/org/apache/hudi/utils/HoodieConfigDocGenerator.java
index 910fffb65790..973098c5d1d3 100644
--- a/hudi-utils/src/main/java/org/apache/hudi/utils/HoodieConfigDocGenerator.java
+++ b/hudi-utils/src/main/java/org/apache/hudi/utils/HoodieConfigDocGenerator.java
@@ -68,7 +68,7 @@ public class HoodieConfigDocGenerator {
"At a high level, you can control behaviour at few levels.";
private static final String BASIC_CONFIGS_PAGE_SUMMARY = "This page covers the basic configurations you may use to " +
"write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a " +
- "full list of all configs, please visit the [All Configurations](/docs/configurations) page.";
+ "full list of all configs, please visit the [All Configurations](configurations.md) page.";
private static final String FLINK_CONFIG_CLASS_NAME = "org.apache.hudi.configuration.FlinkOptions";
private static final String ALL_CONFIGS_PATH = "/tmp/configurations.md";
private static final String BASIC_CONFIGS_PATH = "/tmp/basic_configurations.md";
@@ -151,7 +151,7 @@ public class HoodieConfigDocGenerator {
title: Basic Configurations
summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only
features a subset of the most frequently used configurations. For a full list of all configs, please visit the
- [All Configurations](/docs/configurations) page.
+ [All Configurations](configurations.md) page.
last_modified_at: 2019-12-30T15:59:57-04:00
---
*/
diff --git a/website/versioned_docs/version-0.14.0/basic_configurations.md b/website/versioned_docs/version-0.14.0/basic_configurations.md
index 87c76254bb57..e31ed3380211 100644
--- a/website/versioned_docs/version-0.14.0/basic_configurations.md
+++ b/website/versioned_docs/version-0.14.0/basic_configurations.md
@@ -1,11 +1,11 @@
---
title: Basic Configurations
-summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
last_modified_at: 2023-09-22T14:14:21.715
---
-This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
diff --git a/website/versioned_docs/version-0.14.0/cli.md b/website/versioned_docs/version-0.14.0/cli.md
index 8d44090b7736..2d01102bc70e 100644
--- a/website/versioned_docs/version-0.14.0/cli.md
+++ b/website/versioned_docs/version-0.14.0/cli.md
@@ -578,7 +578,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-0.14.0/cloud.md b/website/versioned_docs/version-0.14.0/cloud.md
index 525c849cd4bc..8c35b4f573a3 100644
--- a/website/versioned_docs/version-0.14.0/cloud.md
+++ b/website/versioned_docs/version-0.14.0/cloud.md
@@ -11,23 +11,23 @@ last_modified_at: 2021-10-12T10:50:00+08:00
Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
to cloud stores.
-* [AWS S3](s3_hoodie) <br/>
+* [AWS S3](s3_hoodie.md) <br/>
Configurations required for S3 and Hudi co-operability.
-* [Google Cloud Storage](gcs_hoodie) <br/>
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
Configurations required for GCS and Hudi co-operability.
-* [Alibaba Cloud OSS](oss_hoodie) <br/>
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
Configurations required for OSS and Hudi co-operability.
-* [Microsoft Azure](azure_hoodie) <br/>
+* [Microsoft Azure](azure_hoodie.md) <br/>
Configurations required for Azure and Hudi co-operability.
-* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
Configurations required for COS and Hudi co-operability.
-* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
Configurations required for IBM Cloud Object Storage and Hudi co-operability.
-* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
Configurations required for BOS and Hudi co-operability.
-* [JuiceFS](jfs_hoodie) <br/>
+* [JuiceFS](jfs_hoodie.md) <br/>
Configurations required for JuiceFS and Hudi co-operability.
-* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
Configurations required for OCI and Hudi co-operability.
:::note
diff --git a/website/versioned_docs/version-0.14.0/clustering.md b/website/versioned_docs/version-0.14.0/clustering.md
index d6c980dce2a1..e668ae8a1f88 100644
--- a/website/versioned_docs/version-0.14.0/clustering.md
+++ b/website/versioned_docs/version-0.14.0/clustering.md
@@ -13,7 +13,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## How is compaction different from clustering?
Hudi is modeled like a log-structured storage engine with multiple versions of the data.
-Particularly, [Merge-On-Read](table_types#merge-on-read-table)
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
@@ -93,7 +93,7 @@ update strategy.
This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
-config [hoodie.clustering.plan.strategy.class](configurations#hoodieclusteringplanstrategyclass).
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
Different plan strategies are as follows:
diff --git a/website/versioned_docs/version-0.14.0/compaction.md b/website/versioned_docs/version-0.14.0/compaction.md
index e986d818fe85..0685d47c5e78 100644
--- a/website/versioned_docs/version-0.14.0/compaction.md
+++ b/website/versioned_docs/version-0.14.0/compaction.md
@@ -13,7 +13,7 @@ not applicable to Copy On Write(COW) tables and only applies to MOR tables.
### Why MOR tables need compaction?
To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
-data is organized in terms of [file groups](file_layouts). Each file group in a MOR table
+data is organized in terms of [file groups](file_layouts.md). Each file group in a MOR table
consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
are appended to log files.
@@ -181,7 +181,7 @@ This model may need a lock provider **if** metadata table is enabled.
:::
#### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
The compactor utility allows to do scheduling and execution of compaction.
Example:
@@ -198,7 +198,7 @@ Note, the `instant-time` parameter is now optional for the Hudi Compactor Utilit
the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
#### Hudi CLI
-Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.14.0/configurations.md b/website/versioned_docs/version-0.14.0/configurations.md
index 20318cacdccc..3fe30572dd12 100644
--- a/website/versioned_docs/version-0.14.0/configurations.md
+++ b/website/versioned_docs/version-0.14.0/configurations.md
@@ -12,7 +12,7 @@ last_modified_at: 2023-09-22T14:14:21.68
This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
:::tip
-With [externalized config file](configurations#externalized-config-file),
+With [externalized config file](configurations.md#externalized-config-file),
instead of directly passing configuration settings to every Hudi job,
you can also centrally set them in a configuration file `hudi-defaults.conf`.
:::
diff --git a/website/versioned_docs/version-0.14.0/deployment.md b/website/versioned_docs/version-0.14.0/deployment.md
index d6f549c000cf..497f870ebb48 100644
--- a/website/versioned_docs/version-0.14.0/deployment.md
+++ b/website/versioned_docs/version-0.14.0/deployment.md
@@ -26,7 +26,7 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### Hudi Streamer
-[Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
To use Hudi Streamer in Spark, the `hudi-utilities-bundle` is required, by adding
@@ -37,7 +37,7 @@ corresponding to the Spark version used, e.g.,
`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.13.0,org.apache.hudi:hudi-spark3.1-bundle_2.12:0.13.0`,
if using `hudi-utilities-bundle` solely in Spark encounters compatibility issues.
- - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data.md#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline [...]
Here is an example invocation using spark datasource
@@ -237,7 +237,7 @@ hoodie.table.version=5
```
Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
-For more details on downgrade, please refer documentation [here](cli#upgrade-and-downgrade-table).
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
```shell
connect --path /tmp/hudi_table
downgrade table --toVersion 4
@@ -258,4 +258,4 @@ Currently migrating to Hudi can be done using two approaches
- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
-For more details, refer to the detailed [migration guide](migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-0.14.0/faq.md b/website/versioned_docs/version-0.14.0/faq.md
index 3fc1742fada5..0b237893e59a 100644
--- a/website/versioned_docs/version-0.14.0/faq.md
+++ b/website/versioned_docs/version-0.14.0/faq.md
@@ -110,7 +110,7 @@ Hudi writers atomically move an inflight write operation to a "completed" state
Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
-* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](https://hudi.apache.org/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond sim [...]
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](https://hudi.apache.org/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond [...]
* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](https://hudi.apache.org/docs/clustering) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously whic [...]
* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudiβs design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
diff --git a/website/versioned_docs/version-0.14.0/file_layouts.md b/website/versioned_docs/version-0.14.0/file_layouts.md
index f0357073890b..80dece85e839 100644
--- a/website/versioned_docs/version-0.14.0/file_layouts.md
+++ b/website/versioned_docs/version-0.14.0/file_layouts.md
@@ -10,8 +10,8 @@ The following describes the general file layout structure for Apache Hudi. Pleas
* Each file group contains several file slices
* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
-Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
-file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner.md) action gets rid of unused/older file slices to reclaim space on the file system.
diff --git a/website/versioned_docs/version-0.14.0/file_sizing.md b/website/versioned_docs/version-0.14.0/file_sizing.md
index 467c7f734a97..2c45dc6d216d 100644
--- a/website/versioned_docs/version-0.14.0/file_sizing.md
+++ b/website/versioned_docs/version-0.14.0/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner.md)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
index 42f15ea2f26a..fde257cb2860 100644
--- a/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
@@ -10,12 +10,12 @@ This page introduces Flink-Hudi integration. We can feel the unique charm of how
This guide helps you quickly start using Flink on Hudi, and learn different modes for reading/writing Hudi by Flink:
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](hoodie_streaming_ingestion#cdc-ingestion), [Bulk Insert](hoodie_streaming_ingestion#bulk-insert), [Index Bootstrap](hoodie_streaming_ingestion#index-bootstrap), [Changelog Mode](hoodie_streaming_ingestion#changelog-mode) and [Append Mode](hoodie_streaming_ingestion#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](querying_data#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
-- **Optimization**: Offline compaction is supported [Offline Compaction](compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](hoodie_streaming_ingestion.md#cdc-ingestion), [Bulk Insert](hoodie_streaming_ingestion.md#bulk-insert), [Index Bootstrap](hoodie_streaming_ingestion.md#index-bootstrap), [Changelog Mode](hoodie_streaming_ingestion.md#changelog-mode) and [Append Mode](hoodie_streaming_ingestion.md#append-mode).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](querying_data.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
## Quick Start
@@ -51,7 +51,7 @@ Start a standalone Flink cluster within hadoop environment.
Before you start up the cluster, we suggest to config the cluster as follows:
- in `$FLINK_HOME/conf/flink-conf.yaml`, add config option `taskmanager.numberOfTaskSlots: 4`
-- in `$FLINK_HOME/conf/flink-conf.yaml`, [add other global configurations according to the characteristics of your task](flink_tuning#global-configurations)
+- in `$FLINK_HOME/conf/flink-conf.yaml`, [add other global configurations according to the characteristics of your task](flink_tuning.md#global-configurations)
- in `$FLINK_HOME/conf/workers`, add item `localhost` as 4 lines so that there are 4 workers on the local cluster
Now starts the cluster:
@@ -66,7 +66,7 @@ export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
#### Step.3 start Flink SQL client
Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
-You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore#install)), or download it from the
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
Now starts the SQL CLI:
@@ -287,7 +287,7 @@ env.execute("Api_Source");
>
This statement queries snapshot view of the dataset.
-Refers to [Table types and queries](concepts#table-types--queries) for more info on all table types and query types supported.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
### Update Data
@@ -305,7 +305,7 @@ UPDATE t1 SET age=19 WHERE uuid in ('id1', 'id2');
```
Notice that the save mode is now `Append`. In general, always use append mode unless you are trying to create the table for the first time.
-[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts)
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
denoted by the timestamp. Look for changes in `_hoodie_commit_time`, `age` fields for the same `_hoodie_record_key`s in previous commit.
:::note
@@ -321,7 +321,7 @@ We do not need to specify endTime, if we want all changes after the given commit
:::note
The bundle jar with **hive profile** is needed for streaming query, by default the officially released flink bundle is built **without**
-**hive profile**, the jar needs to be built manually, see [Build Flink Bundle Jar](syncing_metastore#install) for more details.
+**hive profile**, the jar needs to be built manually, see [Build Flink Bundle Jar](syncing_metastore.md#install) for more details.
:::
```sql
@@ -370,13 +370,13 @@ Only **batch** queries on Hudi table with primary key work correctly.
:::
## Where To Go From Here?
-Check out the [Flink Setup](flink_tuning) how-to page for deeper dive into configuration settings.
+Check out the [Flink Setup](flink_tuning.md) how-to page for deeper dive into configuration settings.
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
- - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- - [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+ - [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -385,5 +385,5 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it. Also, if you are looking for ways to migrate your existing data
-to Hudi, refer to [migration guide](migration_guide).
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-0.14.0/hoodie_cleaner.md b/website/versioned_docs/version-0.14.0/hoodie_cleaner.md
index 311c2c83a312..7921311649da 100644
--- a/website/versioned_docs/version-0.14.0/hoodie_cleaner.md
+++ b/website/versioned_docs/version-0.14.0/hoodie_cleaner.md
@@ -126,7 +126,7 @@ spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls
Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
#### CLI
-You can also use [Hudi CLI](cli) to run Hoodie Cleaner.
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
CLI provides the below commands for cleaner service:
- `cleans show`
diff --git a/website/versioned_docs/version-0.14.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-0.14.0/hoodie_streaming_ingestion.md
index 809c549a55ba..c3f824ef712d 100644
--- a/website/versioned_docs/version-0.14.0/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-0.14.0/hoodie_streaming_ingestion.md
@@ -171,7 +171,7 @@ and then ingest it as follows.
--op BULK_INSERT
```
-In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide).
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
### Note on hudi utilities bundle usage for different spark versions
From 0.11.0 release, we start to provide a new `hudi-utilities-slim-bundle` which aims to exclude dependencies that can
@@ -237,7 +237,7 @@ A Hudi Streamer job can then be triggered as follows:
--op BULK_INSERT
```
-Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control) section
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
## Checkpointing
`HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
@@ -603,7 +603,7 @@ frequent `file handle` switching.
:::note
The parallelism of `bulk_insert` is specified by `write.tasks`. The parallelism will affect the number of small files.
In theory, the parallelism of `bulk_insert` is the number of `bucket`s (In particular, when each bucket writes to maximum file size, it
-will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks).
+will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks).
:::
#### Options
@@ -611,7 +611,7 @@ will rollover to the new file handle. Finally, `the number of files` >= [`write.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.operation` | `true` | `upsert` | Setting as `bulk_insert` to open this function |
-| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks) |
+| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks) |
| `write.bulk_insert.shuffle_input` | `false` | `true` | Whether to shuffle data according to the input field before writing. Enabling this option will reduce the number of small files, but there may be a risk of data skew |
| `write.bulk_insert.sort_input` | `false` | `true` | Whether to sort data according to the input field before writing. Enabling this option will reduce the number of small files when a write task writes multiple partitions |
| `write.sort.memory` | `false` | `128` | Available managed memory of sort operator. default `128` MB |
diff --git a/website/versioned_docs/version-0.14.0/indexing.md b/website/versioned_docs/version-0.14.0/indexing.md
index bd34ca4081cb..448af86f06c4 100644
--- a/website/versioned_docs/version-0.14.0/indexing.md
+++ b/website/versioned_docs/version-0.14.0/indexing.md
@@ -11,9 +11,9 @@ Hudi provides efficient upserts, by mapping a given hoodie key (record key + par
This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
mapped file group contains all versions of a group of records.
-For [Copy-On-Write tables](table_types#copy-on-write-table), this enables fast upsert/delete operations, by
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), this enables fast upsert/delete operations, by
avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](table_types#merge-on-read-table), this design allows Hudi to bound the amount of
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), this design allows Hudi to bound the amount of
records any given base file needs to be merged against.
Specifically, a given base file needs to merged only against updates for records that are part of that base file. In contrast,
designs without an indexing component (e.g: [Apache Hive ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)),
diff --git a/website/versioned_docs/version-0.14.0/metadata.md b/website/versioned_docs/version-0.14.0/metadata.md
index ea6523a1cdc0..e67c01cf8cfe 100644
--- a/website/versioned_docs/version-0.14.0/metadata.md
+++ b/website/versioned_docs/version-0.14.0/metadata.md
@@ -106,7 +106,7 @@ Following are the Spark based basic configs that are needed to enable metadata a
The metadata table with synchronous updates and metadata-table-based file listing are enabled by default.
There are prerequisite configurations and steps in [Deployment considerations](#deployment-considerations-for-metadata-table) to
safely use this feature. The metadata table and related file listing functionality can still be turned off by setting
-[`hoodie.metadata.enable`](configurations#hoodiemetadataenable) to `false`. The
+[`hoodie.metadata.enable`](configurations.md#hoodiemetadataenable) to `false`. The
[multi-modal index](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi) are
disabled by default and can be enabled in write side explicitly using the above configs.
@@ -151,17 +151,17 @@ corresponding configs across Spark and Flink readers.
## Deployment considerations for metadata Table
To ensure that metadata table stays up to date, all write operations on the same Hudi table need additional configurations
besides the above in different deployment models. Before enabling metadata table, all writers on the same table must
-be stopped. Please refer to the different [deployment models](concurrency_control#deployment-models-with-supported-concurrency-controls)
+be stopped. Please refer to the different [deployment models](concurrency_control.md#deployment-models-with-supported-concurrency-controls)
for more details on each model. This section only highlights how to safely enable metadata table in different deployment models.
### Deployment Model A: Single writer with inline table services
-In [Model A](concurrency_control#model-a-single-writer-with-inline-table-services), after setting [`hoodie.metadata.enable`](configurations#hoodiemetadataenable) to `true`, restarting
+In [Model A](concurrency_control.md#model-a-single-writer-with-inline-table-services), after setting [`hoodie.metadata.enable`](configurations.md#hoodiemetadataenable) to `true`, restarting
the single writer is sufficient to safely enable metadata table.
### Deployment Model B: Single writer with async table services
-If your current deployment model is [Model B](concurrency_control#model-b-single-writer-with-async-table-services), enabling metadata
+If your current deployment model is [Model B](concurrency_control.md#model-b-single-writer-with-async-table-services), enabling metadata
table requires adding optimistic concurrency control along with suggested lock provider like below.
```properties
hoodie.write.concurrency.mode=optimistic_concurrency_control
@@ -179,7 +179,7 @@ process which cannot rely on the in-process lock provider.
### Deployment Model C: Multi-writer
-If your current deployment model is [multi-writer](concurrency_control#model-c-multi-writer) along with a lock
+If your current deployment model is [multi-writer](concurrency_control.md#model-c-multi-writer) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only partial writers leads to loss of data from the inconsistent writer. So, ensure you enable
@@ -190,5 +190,5 @@ hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.write.lock.provider=<distributed-lock-provider-classname>
```
-Note that there are different external [lock providers available](concurrency_control#external-locking-and-lock-providers)
+Note that there are different external [lock providers available](concurrency_control.md#external-locking-and-lock-providers)
to choose from.
diff --git a/website/versioned_docs/version-0.14.0/metadata_indexing.md b/website/versioned_docs/version-0.14.0/metadata_indexing.md
index 43669dbca59e..6a9c901ee804 100644
--- a/website/versioned_docs/version-0.14.0/metadata_indexing.md
+++ b/website/versioned_docs/version-0.14.0/metadata_indexing.md
@@ -5,7 +5,7 @@ toc: true
last_modified_at:
---
-Hudi maintains a scalable [metadata](metadata) that has some auxiliary data about the table.
+Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data about the table.
The [pluggable indexing subsystem](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi)
of Hudi depends on the metadata table. Different types of index, from `files` index for locating records efficiently
to `column_stats` index for data skipping, are part of the metadata table. A fundamental tradeoff in any data system
@@ -25,7 +25,7 @@ feature, please check out [this blog](https://www.onehouse.ai/blog/asynchronous-
## Setup Async Indexing
-First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) which will continuously write data
+First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) which will continuously write data
from raw parquet to Hudi table. We used the widely available [NY Taxi dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/), whose setup details are as below:
<details>
<summary>Ingestion write config</summary>
@@ -78,7 +78,7 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](metadata#metadata-table-indices) along with [configs](metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
+metadata table can be explored [here](metadata.md#metadata-table-indices) along with [configs](metadata.md#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
diff --git a/website/versioned_docs/version-0.14.0/metrics.md b/website/versioned_docs/version-0.14.0/metrics.md
index 38152a31ef64..95a18fcdba68 100644
--- a/website/versioned_docs/version-0.14.0/metrics.md
+++ b/website/versioned_docs/version-0.14.0/metrics.md
@@ -6,7 +6,7 @@ toc: true
last_modified_at: 2020-06-20T15:59:57-04:00
---
-In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations#METRICS).
+In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations.md#METRICS).
## MetricsReporter
@@ -17,7 +17,7 @@ MetricsReporter provides APIs for reporting `HoodieMetrics` to user-specified ba
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
#### Configurations
-The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Jmx).
+The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Jmx).
```properties
hoodie.metrics.on=true
@@ -37,7 +37,7 @@ As configured above, JmxMetricsReporter will started JMX server on port 4001. We
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send `HoodieMetrics` to it.
#### Configurations
-The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations#Metrics-Configurations-for-Graphite).
+The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Graphite).
```properties
hoodie.metrics.on=true
@@ -58,7 +58,7 @@ DatadogMetricsReporter is an implementation of Datadog reporter.
A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
#### Configurations
-The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Datadog-reporter).
+The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Datadog-reporter).
```properties
hoodie.metrics.on=true
@@ -138,7 +138,7 @@ tuned are in the `HoodieMetricsCloudWatchConfig` class.
Allows users to define a custom metrics reporter.
#### Configurations
-The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations).
+The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations).
```properties
hoodie.metrics.on=true
diff --git a/website/versioned_docs/version-0.14.0/migration_guide.md b/website/versioned_docs/version-0.14.0/migration_guide.md
index ef0d49ef2153..b2b62550c9ad 100644
--- a/website/versioned_docs/version-0.14.0/migration_guide.md
+++ b/website/versioned_docs/version-0.14.0/migration_guide.md
@@ -40,7 +40,7 @@ There are a few options when choosing this approach.
#### Using Hudi Streamer
-Use the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool. HoodieStreamer supports bootstrap with
+Use the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool. HoodieStreamer supports bootstrap with
--run-bootstrap command line option. There are two types of bootstrap, METADATA_ONLY and FULL_RECORD. METADATA_ONLY will
generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will
perform a full copy/rewrite of the data as a Hudi table. Additionally, once can choose selective partitions using regex
@@ -88,7 +88,7 @@ Refer to [Bootstrap procedure](https://hudi.apache.org/docs/procedures#bootstrap
#### Using Hudi CLI
Write your own custom logic of how to load an existing table into a Hudi managed one. Please read about the RDD API
-[here](quick-start-guide). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
+[here](quick-start-guide.md). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hudi-cli && ./hudi-cli.sh`.
```java
diff --git a/website/versioned_docs/version-0.14.0/overview.mdx b/website/versioned_docs/version-0.14.0/overview.mdx
index 6099f9b43a36..8f8a8ffd544d 100644
--- a/website/versioned_docs/version-0.14.0/overview.mdx
+++ b/website/versioned_docs/version-0.14.0/overview.mdx
@@ -13,27 +13,27 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced βhoodieβ) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
-Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](sql_ddl),
-[transactions](timeline), [efficient upserts/deletes](write_operations), [advanced indexes](indexing),
-[streaming ingestion services](hoodie_streaming_ingestion), data [clustering](clustering)/[compaction](compaction) optimizations,
-and [concurrency](concurrency_control) all while keeping your data in open source file formats.
+Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](sql_ddl.md),
+[transactions](timeline.md), [efficient upserts/deletes](write_operations.md), [advanced indexes](indexing.md),
+[streaming ingestion services](hoodie_streaming_ingestion.md), data [clustering](clustering.md)/[compaction](compaction.md) optimizations,
+and [concurrency](concurrency_control.md) all while keeping your data in open source file formats.
Not only is Apache Hudi great for streaming workloads, but it also allows you to create efficient incremental batch pipelines.
-Read the docs for more [use case descriptions](use_cases) and check out [who's using Hudi](/powered-by), to see how some of the
+Read the docs for more [use case descriptions](use_cases.md) and check out [who's using Hudi](/powered-by), to see how some of the
largest data lakes in the world including [Uber](https://www.uber.com/en-IN/blog/uber-big-data-platform/), [Amazon](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/),
[ByteDance](http://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance),
[Robinhood](https://s.apache.org/hudi-robinhood-talk) and more are transforming their production data lakes with Hudi.
-Apache Hudi can easily be used on any [cloud storage platform](cloud).
+Apache Hudi can easily be used on any [cloud storage platform](cloud.md).
Hudiβs advanced performance optimizations, make analytical workloads faster with any of
the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
-- [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
-- [Hudi File Layout](file_layouts) - How the files are laid out on storage
-- [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
-- [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+- [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+- [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+- [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -41,11 +41,11 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
## Getting Started
Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:
-- [Spark Quick Start Guide](quick-start-guide) β if you primarily use Apache Spark
-- [Flink Quick Start Guide](flink-quick-start-guide) β if you primarily use Apache Flink
+- [Spark Quick Start Guide](quick-start-guide.md) β if you primarily use Apache Spark
+- [Flink Quick Start Guide](flink-quick-start-guide.md) β if you primarily use Apache Flink
If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the Docker Demo:
-- [Docker Demo](docker_demo)
+- [Docker Demo](docker_demo.md)
## Connect With The Community
Apache Hudi is community focused and community led and welcomes new-comers with open arms. Leverage the following
diff --git a/website/versioned_docs/version-0.14.0/performance.md b/website/versioned_docs/version-0.14.0/performance.md
index cd9de72d5a08..9fec829e492f 100644
--- a/website/versioned_docs/version-0.14.0/performance.md
+++ b/website/versioned_docs/version-0.14.0/performance.md
@@ -16,12 +16,12 @@ Here are some ways to efficiently manage the storage of your Hudi tables.
- The [small file handling feature](configurations/#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](configurations#hoodieparquetcompressionratio),
+- Cleaner can be [configured](configurations.md#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](configurations.md#hoodieparquetmaxfilesize), [log files](configurations.md#hoodielogfilemaxsize) & expected [compression ratio](configurations.md#hoodieparquetcompressionratio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
-- Intelligently tuning the [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
+- Intelligently tuning the [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be changed without re-clustering the table. Writes will simply expand given file groups with new updates/inserts as explained before.
-- For workloads with heavy updates, the [merge-on-read table](concepts#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
+- For workloads with heavy updates, the [merge-on-read table](concepts.md#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
## Performance Optimizations
@@ -119,11 +119,11 @@ For very large tables (1Tb+, 10s of 1000s of files), Data skipping could
To unlock the power of Data Skipping you will need to
-1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
+1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing.md)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
2. Enable Data Skipping in your queries, using `hoodie.metadata.index.column.stats.enable=true` (to enable Column Stats Index being populated on the write path, disabled by default)
:::note
-If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
+If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing.md) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
:::
To enable Data Skipping in your queries make sure to set following properties to `true` (on the read path):
diff --git a/website/versioned_docs/version-0.14.0/precommit_validator.md b/website/versioned_docs/version-0.14.0/precommit_validator.md
index 1d067b202d3e..645cfce6abd0 100644
--- a/website/versioned_docs/version-0.14.0/precommit_validator.md
+++ b/website/versioned_docs/version-0.14.0/precommit_validator.md
@@ -8,7 +8,7 @@ Data quality refers to the overall accuracy, completeness, consistency, and vali
Hudi offers **Pre-Commit Validators** that allow you to ensure that your data meets certain data quality expectations as you are writing with Hudi Streamer or Spark Datasource writers.
:::note
-Pre-commit validators are skipped when using the [BULK_INSERT](write_operations#bulk_insert) write operation type.
+Pre-commit validators are skipped when using the [BULK_INSERT](write_operations.md#bulk_insert) write operation type.
:::
Multiple class names can be separated by `,` delimiter.
diff --git a/website/versioned_docs/version-0.14.0/procedures.md b/website/versioned_docs/version-0.14.0/procedures.md
index 8940c078667b..b82fec4650b5 100644
--- a/website/versioned_docs/version-0.14.0/procedures.md
+++ b/website/versioned_docs/version-0.14.0/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations.md#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations.md#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations.md#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations.md#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1473,11 +1473,11 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
-| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
-| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [clean_policy](configurations.md#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum qu [...]
+| [retain_commits](configurations.md#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations.md#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of [...]
+| [file_versions_retained](configurations.md#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations.md#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
@@ -1573,12 +1573,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations.md#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations.md#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations.md#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations.md#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations.md#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations.md#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1788,18 +1788,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations.md#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations.md#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations.md#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
+| [parallelism](configurations.md#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-0.14.0/querying_data.md b/website/versioned_docs/version-0.14.0/querying_data.md
index b0102fb2b335..4cc1d3921034 100644
--- a/website/versioned_docs/version-0.14.0/querying_data.md
+++ b/website/versioned_docs/version-0.14.0/querying_data.md
@@ -7,10 +7,10 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl.md), [SQL DML](sql_dml.md), [SQL Queries](sql_queries.md) and [Procedures](procedures.md) for the latest documentation.
:::
-Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts#query-types).
+Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts.md#query-types).
Once the table is synced to the Hive metastore, it provides external Hive tables backed by Hudi's custom inputformats. Once the proper hudi
bundle has been installed, the table can be queried by popular query engines like Hive, Spark SQL, Flink, Trino and PrestoDB.
@@ -19,7 +19,7 @@ In sections, below we will discuss specific setup to access different query type
## Spark Datasource
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
-See the [Spark Quick Start](quick-start-guide) for more examples of Spark datasource reading queries.
+See the [Spark Quick Start](quick-start-guide.md) for more examples of Spark datasource reading queries.
**Setup**
@@ -56,8 +56,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Incremental Queries](quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](configurations#SPARK_DATASOURCE) section, to view all datasource options.
+For examples, refer to [Incremental Queries](quick-start-guide.md#incremental-query) in the Spark quickstart.
+Please refer to [configurations](configurations.md#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.14.0/quick-start-guide.md b/website/versioned_docs/version-0.14.0/quick-start-guide.md
index e98ff6bc1d3e..54ea6f908666 100644
--- a/website/versioned_docs/version-0.14.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.0/quick-start-guide.md
@@ -262,7 +262,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl.md) reference guide.
</TabItem>
@@ -306,7 +306,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -339,7 +339,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -348,7 +348,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml.md#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -460,7 +460,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml.md#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -490,7 +490,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts.md).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -578,7 +578,7 @@ MERGE statement either using `SET *` or using `SET column1 = expression1 [, colu
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data.md#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -948,7 +948,7 @@ Note that CDC queries are currently only supported on Copy-on-Write tables.
The examples thus far have showcased one of the two table types, that Hudi supports - Copy-on-Write (COW) tables. Hudi also supports
a more advanced write-optimized table type called Merge-on-Read (MOR) tables, that can balance read and write performance in a more
-flexible manner. See [table types](table_types) for more details.
+flexible manner. See [table types](table_types.md) for more details.
Any of these examples can be run on a Merge-on-Read table by simply changing the table type to MOR, while creating the table, as below.
@@ -1074,7 +1074,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations.md).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1147,29 +1147,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide.md).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
-For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl.md) and [Spark SQL DML](sql_dml.md) reference guides.
+For alter table commands, check out [this](sql_ddl.md#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures.md) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion.md#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](hoodie_streaming_ingestion#structured-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](hoodie_streaming_ingestion.md#structured-streaming) for more.
-Check out more information on [modeling data in Hudi](faq#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](writing_data).
+Check out more information on [modeling data in Hudi](faq.md#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](writing_data.md).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it.
+steps [here](docker_demo.md) to get a taste for it.
diff --git a/website/versioned_docs/version-0.14.0/record_payload.md b/website/versioned_docs/version-0.14.0/record_payload.md
index a1fd92806079..f236660ba878 100644
--- a/website/versioned_docs/version-0.14.0/record_payload.md
+++ b/website/versioned_docs/version-0.14.0/record_payload.md
@@ -105,7 +105,7 @@ hoodie.datasource.write.payload.class=org.apache.hudi.common.model.EventTimeAvro
```
This is the default record payload for Flink based writing. Some use cases require merging records by event time and
thus event time plays the role of an ordering field. This payload is particularly useful in the case of late-arriving data.
-For such use cases, users need to set the [payload event time field](configurations#RECORD_PAYLOAD) configuration.
+For such use cases, users need to set the [payload event time field](configurations.md#RECORD_PAYLOAD) configuration.
#### OverwriteNonDefaultsWithLatestAvroPayload
```scala
@@ -171,6 +171,6 @@ provides support for seamlessly applying changes captured via Debezium for MySQL
provides support for applying changes captured via Amazon Database Migration Service onto S3.
Record payloads are tunable to suit many use cases. Please check out the configurations
-listed [here](configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
+listed [here](configurations.md#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
please check out [this FAQ](faq/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.14.0/s3_hoodie.md b/website/versioned_docs/version-0.14.0/s3_hoodie.md
index 5faad6e62be9..fe3c9e513cfe 100644
--- a/website/versioned_docs/version-0.14.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.14.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](hoodie_cleaner.md) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.14.0/sql_ddl.md b/website/versioned_docs/version-0.14.0/sql_ddl.md
index 1cc65e5b07d2..b08e916c243f 100644
--- a/website/versioned_docs/version-0.14.0/sql_ddl.md
+++ b/website/versioned_docs/version-0.14.0/sql_ddl.md
@@ -75,7 +75,7 @@ should be specified as `PARTITIONED BY (dt, hh)`.
### Create table with record keys and ordering fields
-As discussed [here](quick-start-guide#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
Typically, this is also accompanied by configuring a `preCombineField` option to deal with out-of-order data and potential
duplicate records with the same key in the incoming writes.
@@ -104,7 +104,7 @@ TBLPROPERTIES (
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion.md#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -240,7 +240,7 @@ Users can set table properties while creating a table. The important table prope
| Parameter Name | Default | Description |
|------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types) |
+| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types.md) |
| primaryKey | uuid | The primary key field names of the table separated by commas. Same as `hoodie.datasource.write.recordkey.field`. If this config is ignored, hudi will auto-generate primary keys. If explicitly set, primary key generation will honor user configuration. |
| preCombineField | | The pre-combine field of the table. It is used for resolving the final version of the record among multiple versions. Generally, `event time` or another similar column will be used for ordering purposes. Hudi will be able to handle out-of-order data using the preCombine field value. |
@@ -392,7 +392,7 @@ CREATE CATALOG hoodie_catalog
### Create Table
-The following is an example of creating a Flink table. Read the [Flink Quick Start](flink-quick-start-guide) guide for more examples.
+The following is an example of creating a Flink table. Read the [Flink Quick Start](flink-quick-start-guide.md) guide for more examples.
```sql
CREATE TABLE hudi_table2(
diff --git a/website/versioned_docs/version-0.14.0/sql_dml.md b/website/versioned_docs/version-0.14.0/sql_dml.md
index 4508560e2131..5cbb96379417 100644
--- a/website/versioned_docs/version-0.14.0/sql_dml.md
+++ b/website/versioned_docs/version-0.14.0/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
### Insert Into
@@ -25,7 +25,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -45,7 +45,7 @@ INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000
```
:::note Mapping to write operations
-Hudi offers flexibility in choosing the underlying [write operation](write_operations) of a `INSERT INTO` statement using
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
and *"upsert"* (with deduplication/merging). If a precombine field is not set, *"insert"* is chosen as the default. For a table with preCombine field set,
*"upsert"* is chosen as the default operation.
@@ -200,6 +200,6 @@ You can control the behavior of these operations using various configuration opt
## Flink
Flink SQL also provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. All these operations are already
-showcased in the [Flink Quickstart](flink-quick-start-guide).
+showcased in the [Flink Quickstart](flink-quick-start-guide.md).
diff --git a/website/versioned_docs/version-0.14.0/sql_queries.md b/website/versioned_docs/version-0.14.0/sql_queries.md
index e38ff1eb9994..df284395ec6e 100644
--- a/website/versioned_docs/version-0.14.0/sql_queries.md
+++ b/website/versioned_docs/version-0.14.0/sql_queries.md
@@ -8,11 +8,11 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-Hudi stores and organizes data on storage while providing different ways of [querying](concepts#query-types), across a wide range of query engines.
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
This page will show how to issue different queries and discuss any specific instructions for each query engine.
## Spark SQL
-The Spark [quickstart](quick-start-guide) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
@@ -91,11 +91,11 @@ For e.g the table has received 10 million modifications across 1 million records
see all changes in a given time window and not just the latest values.
:::
-Please refer to [configurations](basic_configurations) section for the important configuration options.
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
## Flink SQL
Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
-relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide).
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-0.14.0/structure.md b/website/versioned_docs/version-0.14.0/structure.md
index 31281b68f71c..9f78f89f16f5 100644
--- a/website/versioned_docs/version-0.14.0/structure.md
+++ b/website/versioned_docs/version-0.14.0/structure.md
@@ -17,4 +17,4 @@ Hudi (pronounced βHoodieβ) ingests & manages storage of large analytical tab
By carefully managing how data is laid out in storage & how itβs exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [PrestoDB](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) [...]
-Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide) for a demo.
+Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide.md) for a demo.
diff --git a/website/versioned_docs/version-0.14.0/syncing_aws_glue_data_catalog.md b/website/versioned_docs/version-0.14.0/syncing_aws_glue_data_catalog.md
index e54c6d528876..b625c36ef3f5 100644
--- a/website/versioned_docs/version-0.14.0/syncing_aws_glue_data_catalog.md
+++ b/website/versioned_docs/version-0.14.0/syncing_aws_glue_data_catalog.md
@@ -10,7 +10,7 @@ and send them to AWS Glue.
### Configurations
There is no additional configuration for using `AwsGlueCatalogSyncTool`; you just need to set it as one of the sync tool
-classes for `HoodieStreamer` and everything configured as shown in [Sync to Hive Metastore](syncing_metastore) will
+classes for `HoodieStreamer` and everything configured as shown in [Sync to Hive Metastore](syncing_metastore.md) will
be passed along.
```shell
diff --git a/website/versioned_docs/version-0.14.0/syncing_metastore.md b/website/versioned_docs/version-0.14.0/syncing_metastore.md
index ed427195f368..4417cd3d9124 100644
--- a/website/versioned_docs/version-0.14.0/syncing_metastore.md
+++ b/website/versioned_docs/version-0.14.0/syncing_metastore.md
@@ -12,7 +12,7 @@ table to Hive metastore.
## Hive Sync Tool
-Writing data with [DataSource](writing_data) writer or [HoodieStreamer](hoodie_streaming_ingestion) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
+Writing data with [DataSource](writing_data.md) writer or [HoodieStreamer](hoodie_streaming_ingestion.md) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
In case, it's preferable to run this from commandline or in an independent jvm, Hudi provides a `HiveSyncTool`, which can be invoked as below,
once you have built the hudi-hive module. Following is how we sync the above Datasource Writer written table to Hive metastore.
diff --git a/website/versioned_docs/version-0.14.0/use_cases.md b/website/versioned_docs/version-0.14.0/use_cases.md
index fb6061b8d1b5..61ea8ccddb8a 100644
--- a/website/versioned_docs/version-0.14.0/use_cases.md
+++ b/website/versioned_docs/version-0.14.0/use_cases.md
@@ -85,7 +85,7 @@ stream processing world to ensure pipelines don't break from non backwards compa
### ACID Transactions
Along with a table, Apache Hudi brings ACID transactional guarantees to a data lake.
-Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline), stamped with an
+Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline.md), stamped with an
instant time that denotes the time at which the action
is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between writer processes
(that issue userβs upserts/deletes), table services (that write data/metadata to optimize/perform bookkeeping) and readers
@@ -127,12 +127,12 @@ cost savings for your data lake.
Some examples of the Apache Hudi services that make this performance optimization easy include:
-- [Auto File Sizing](file_sizing) - to solve the "small files" problem.
-- [Clustering](clustering) - to co-locate data next to each other.
-- [Compaction](compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](indexing) - for efficient upserts and deletes.
+- [Auto File Sizing](file_sizing.md) - to solve the "small files" problem.
+- [Clustering](clustering.md) - to co-locate data next to each other.
+- [Compaction](compaction.md) - to allow tuning of low latency ingestion and fast read queries.
+- [Indexing](indexing.md) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner.md) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.14.0/write_operations.md b/website/versioned_docs/version-0.14.0/write_operations.md
index 71f797d4b54c..d72ee1a9b58a 100644
--- a/website/versioned_docs/version-0.14.0/write_operations.md
+++ b/website/versioned_docs/version-0.14.0/write_operations.md
@@ -6,7 +6,7 @@ last_modified_at:
---
It may be helpful to understand the different write operations of Hudi and how best to leverage them. These operations
-can be chosen/changed across each commit/deltacommit issued against the table. See the [How To docs on Writing Data](writing_data)
+can be chosen/changed across each commit/deltacommit issued against the table. See the [How To docs on Writing Data](writing_data.md)
to see more examples.
## Operation Types
@@ -86,24 +86,24 @@ Here are thh basic configs relevant to the write operations types mentioned abov
## Writing path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](configurations#hoodiecombinebeforeinsert)
+1. [Deduping](configurations.md#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](indexing)
+2. [Index Lookup](indexing.md)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](file_sizing)
+3. [File Sizing](file_sizing.md)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](file_layouts)
+4. [Partitioning](file_layouts.md)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](indexing)
+6. Update [Index](indexing.md)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](hoodie_cleaner) (if needed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data.md#commit-notifications) is exposed)
+8. [Clean](hoodie_cleaner.md) (if needed)
1. Following the commit, cleaning is invoked if needed.
-9. [Compaction](compaction)
+9. [Compaction](compaction.md)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
10. Archive
- 1. Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
+ 1. Lastly, we perform an archival step which moves old [timeline](timeline.md) items to an archive folder.
diff --git a/website/versioned_docs/version-0.14.0/writing_data.md b/website/versioned_docs/version-0.14.0/writing_data.md
index 6a9d0b901d03..0ccc1c8b5689 100644
--- a/website/versioned_docs/version-0.14.0/writing_data.md
+++ b/website/versioned_docs/version-0.14.0/writing_data.md
@@ -9,7 +9,7 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
In this section, we will cover ways to ingest new changes from external sources or even other Hudi tables.
-The two main tools available are the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool, as well as the [Spark Hudi datasource](#spark-datasource-writer).
+The two main tools available are the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool, as well as the [Spark Hudi datasource](#spark-datasource-writer).
## Spark Datasource Writer
@@ -31,15 +31,15 @@ Default value: `"partitionpath"`<br/>
**PRECOMBINE_FIELD_OPT_KEY** (Required): When two records within the same batch have the same key value, the record with the largest value from the field specified will be choosen. If you are using default payload of OverwriteWithLatestAvroPayload for HoodieRecordPayload (`WRITE_PAYLOAD_CLASS`), an incoming record will always takes precendence compared to the one in storage ignoring this `PRECOMBINE_FIELD_OPT_KEY`. <br/>
Default value: `"ts"`<br/>
-**OPERATION_OPT_KEY**: The [write operations](write_operations) to use.<br/>
+**OPERATION_OPT_KEY**: The [write operations](write_operations.md) to use.<br/>
Available values:<br/>
`UPSERT_OPERATION_OPT_VAL` (default), `BULK_INSERT_OPERATION_OPT_VAL`, `INSERT_OPERATION_OPT_VAL`, `DELETE_OPERATION_OPT_VAL`
-**TABLE_TYPE_OPT_KEY**: The [type of table](concepts#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
+**TABLE_TYPE_OPT_KEY**: The [type of table](concepts.md#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
Available values:<br/>
-[`COW_TABLE_TYPE_OPT_VAL`](concepts#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts#merge-on-read-table)
+[`COW_TABLE_TYPE_OPT_VAL`](concepts.md#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts.md#merge-on-read-table)
-**KEYGENERATOR_CLASS_OPT_KEY**: Refer to [Key Generation](key_generation) section below.
+**KEYGENERATOR_CLASS_OPT_KEY**: Refer to [Key Generation](key_generation.md) section below.
**HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY**: If using hive, specify if the table should or should not be partitioned.<br/>
Available values:<br/>
@@ -91,9 +91,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](https://hudi.apache.org/learn/faq/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -127,9 +127,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](https://hudi.apache.org/learn/faq/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
diff --git a/website/versioned_docs/version-0.14.1/basic_configurations.md b/website/versioned_docs/version-0.14.1/basic_configurations.md
index 004e99050be2..7e3c63623410 100644
--- a/website/versioned_docs/version-0.14.1/basic_configurations.md
+++ b/website/versioned_docs/version-0.14.1/basic_configurations.md
@@ -1,11 +1,11 @@
---
title: Basic Configurations
-summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
last_modified_at: 2024-01-03T18:52:29.439
---
-This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
- [**Flink Sql Configs**](#FLINK_SQL): These configs control the Hudi Flink SQL source/sink connectors, providing ability to define record keys, pick out the write operation, specify how to merge records, enable/disable asynchronous compaction or choosing query type to read.
diff --git a/website/versioned_docs/version-0.14.1/cli.md b/website/versioned_docs/version-0.14.1/cli.md
index 13fd9c0ac1ad..77bc50102bba 100644
--- a/website/versioned_docs/version-0.14.1/cli.md
+++ b/website/versioned_docs/version-0.14.1/cli.md
@@ -578,7 +578,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-0.14.1/cloud.md b/website/versioned_docs/version-0.14.1/cloud.md
index 525c849cd4bc..8c35b4f573a3 100644
--- a/website/versioned_docs/version-0.14.1/cloud.md
+++ b/website/versioned_docs/version-0.14.1/cloud.md
@@ -11,23 +11,23 @@ last_modified_at: 2021-10-12T10:50:00+08:00
Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
to cloud stores.
-* [AWS S3](s3_hoodie) <br/>
+* [AWS S3](s3_hoodie.md) <br/>
Configurations required for S3 and Hudi co-operability.
-* [Google Cloud Storage](gcs_hoodie) <br/>
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
Configurations required for GCS and Hudi co-operability.
-* [Alibaba Cloud OSS](oss_hoodie) <br/>
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
Configurations required for OSS and Hudi co-operability.
-* [Microsoft Azure](azure_hoodie) <br/>
+* [Microsoft Azure](azure_hoodie.md) <br/>
Configurations required for Azure and Hudi co-operability.
-* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
Configurations required for COS and Hudi co-operability.
-* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
Configurations required for IBM Cloud Object Storage and Hudi co-operability.
-* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
Configurations required for BOS and Hudi co-operability.
-* [JuiceFS](jfs_hoodie) <br/>
+* [JuiceFS](jfs_hoodie.md) <br/>
Configurations required for JuiceFS and Hudi co-operability.
-* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
Configurations required for OCI and Hudi co-operability.
:::note
diff --git a/website/versioned_docs/version-0.14.1/clustering.md b/website/versioned_docs/version-0.14.1/clustering.md
index 6bda191edc9d..1fba26b2701c 100644
--- a/website/versioned_docs/version-0.14.1/clustering.md
+++ b/website/versioned_docs/version-0.14.1/clustering.md
@@ -13,7 +13,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## How is compaction different from clustering?
Hudi is modeled like a log-structured storage engine with multiple versions of the data.
-Particularly, [Merge-On-Read](table_types#merge-on-read-table)
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
@@ -93,7 +93,7 @@ update strategy.
This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
-config [hoodie.clustering.plan.strategy.class](configurations#hoodieclusteringplanstrategyclass).
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
Different plan strategies are as follows:
diff --git a/website/versioned_docs/version-0.14.1/compaction.md b/website/versioned_docs/version-0.14.1/compaction.md
index 04e5aa773ab4..c50aacd7d49c 100644
--- a/website/versioned_docs/version-0.14.1/compaction.md
+++ b/website/versioned_docs/version-0.14.1/compaction.md
@@ -13,7 +13,7 @@ not applicable to Copy On Write(COW) tables and only applies to MOR tables.
### Why MOR tables need compaction?
To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
-data is organized in terms of [file groups](file_layouts). Each file group in a MOR table
+data is organized in terms of [file groups](file_layouts.md). Each file group in a MOR table
consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
are appended to log files.
@@ -183,7 +183,7 @@ This model may need a lock provider **if** metadata table is enabled.
:::
#### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
The compactor utility allows to do scheduling and execution of compaction.
Example:
@@ -200,7 +200,7 @@ Note, the `instant-time` parameter is now optional for the Hudi Compactor Utilit
the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
#### Hudi CLI
-Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.14.1/concurrency_control.md b/website/versioned_docs/version-0.14.1/concurrency_control.md
index 3efcc0492494..5d3a6b3ad3d0 100644
--- a/website/versioned_docs/version-0.14.1/concurrency_control.md
+++ b/website/versioned_docs/version-0.14.1/concurrency_control.md
@@ -77,7 +77,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](writing_data#non-blocking-concurrency-control-experimental).
+writers [here](writing_data.md#non-blocking-concurrency-control-experimental).
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.14.1/deployment.md b/website/versioned_docs/version-0.14.1/deployment.md
index d6f549c000cf..497f870ebb48 100644
--- a/website/versioned_docs/version-0.14.1/deployment.md
+++ b/website/versioned_docs/version-0.14.1/deployment.md
@@ -26,7 +26,7 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### Hudi Streamer
-[Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
To use Hudi Streamer in Spark, the `hudi-utilities-bundle` is required, by adding
@@ -37,7 +37,7 @@ corresponding to the Spark version used, e.g.,
`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.13.0,org.apache.hudi:hudi-spark3.1-bundle_2.12:0.13.0`,
if using `hudi-utilities-bundle` solely in Spark encounters compatibility issues.
- - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data.md#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline [...]
Here is an example invocation using spark datasource
@@ -237,7 +237,7 @@ hoodie.table.version=5
```
Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
-For more details on downgrade, please refer documentation [here](cli#upgrade-and-downgrade-table).
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
```shell
connect --path /tmp/hudi_table
downgrade table --toVersion 4
@@ -258,4 +258,4 @@ Currently migrating to Hudi can be done using two approaches
- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
-For more details, refer to the detailed [migration guide](migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-0.14.1/faq.md b/website/versioned_docs/version-0.14.1/faq.md
index 3c0a6dd3fda3..e0d1242c97cd 100644
--- a/website/versioned_docs/version-0.14.1/faq.md
+++ b/website/versioned_docs/version-0.14.1/faq.md
@@ -6,10 +6,10 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](faq_general)
-- [Design & Concepts](faq_design_and_concepts)
-- [Writing Tables](faq_writing_tables)
-- [Querying Tables](faq_querying_tables)
-- [Table Services](faq_table_services)
-- [Storage](faq_storage)
-- [Integrations](faq_integrations)
+- [General](faq_general.md)
+- [Design & Concepts](faq_design_and_concepts.md)
+- [Writing Tables](faq_writing_tables.md)
+- [Querying Tables](faq_querying_tables.md)
+- [Table Services](faq_table_services.md)
+- [Storage](faq_storage.md)
+- [Integrations](faq_integrations.md)
diff --git a/website/versioned_docs/version-0.14.1/faq_design_and_concepts.md b/website/versioned_docs/version-0.14.1/faq_design_and_concepts.md
index 720f4ae8e2df..8aee10defc1a 100644
--- a/website/versioned_docs/version-0.14.1/faq_design_and_concepts.md
+++ b/website/versioned_docs/version-0.14.1/faq_design_and_concepts.md
@@ -6,14 +6,14 @@ keywords: [hudi, writing, reading]
### How does Hudi ensure atomicity?
-Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
+Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline.md) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
### Does Hudi extend the Hive table layout?
Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
-* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
-* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users exper [...]
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
+* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering.md) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users ex [...]
* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudiβs design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
### What concurrency control approaches does Hudi adopt?
@@ -32,7 +32,7 @@ In these scenarios, it might be tempting to think of data inconsistencies/data l
(examples [1](https://github.com/apache/hudi/blob/aea5bb6f0ab824247f5e3498762ad94f643a2cb6/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L76),
[2](https://github.com/apache/hudi/blame/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L173)) in incremental queries to ensure that no data
is served beyond the point there is an inflight instant in its timeline, so no data loss or drop happens. This detection is made possible because Hudi writes first request a transaction on the timeline, before planning/executing
-the write, as explained in the [timeline](timeline#states) section.
+the write, as explained in the [timeline](timeline.md#states) section.
In this case, on seeing C1βs inflight commit (publish to timeline is atomic), C2 data (which is > C1 in the timeline) is not served until C1 inflight transitions to a terminal state such as completed or marked as failed.
This [test](https://github.com/apache/hudi/blob/master/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java#L137) demonstrates how Hudi incremental source stops proceeding until C1 completes.
@@ -48,7 +48,7 @@ To expand more on the long term approach, Hudi has had a proposal to streamline/
This has been delayed for a few reasons
- Large hosted query engines and users not upgrading fast enough.
-- The issues brought up - \[[1](faq_design_and_concepts#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
+- The issues brought up - \[[1](faq_design_and_concepts.md#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts.md#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
relevant to this are not practically very important to users beyond good pedantic discussions,
- Wanting to do it alongside [non-blocking concurrency control](https://github.com/apache/hudi/pull/7907) in Hudi version 1.x.
diff --git a/website/versioned_docs/version-0.14.1/faq_general.md b/website/versioned_docs/version-0.14.1/faq_general.md
index c3949365916b..d84cdc4b4946 100644
--- a/website/versioned_docs/version-0.14.1/faq_general.md
+++ b/website/versioned_docs/version-0.14.1/faq_general.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### When is Hudi useful for me or my organization?
-If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi can provide you tools to [help](writing_data/). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
+If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi can provide you tools to [help](writing_data.md). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
@@ -37,7 +37,7 @@ Choose Copy-on-write storage if :
* You are looking for a simple alternative, that replaces your existing parquet tables without any need for real-time data.
* Your current job is rewriting entire table/partition to deal with updates, while only a few files actually change in each partition.
-* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
+* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations.md#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
* Your workload is fairly well-understood and does not have sudden bursts of large amount of update or inserts to older partitions. COW absorbs all the merging cost on the writer side and thus these sudden changes can clog up your ingestion and interfere with meeting normal mode ingest latency targets.
Choose merge-on-read storage if :
@@ -51,7 +51,7 @@ Immaterial of what you choose, Hudi provides
* Incremental pulls
* Ability to de-duplicate data
-Find more [here](concepts/).
+Find more [here](concepts.md).
### Is Hudi an analytical database?
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data/) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data.md) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
@@ -79,7 +79,7 @@ generate the key internally (much like RDBMS row\_ids)
### How does Hudi actually store data inside a table?
-At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts/) for more details.
+At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts.md) for more details.
### How Hudi handles partition evolution requirements ?
diff --git a/website/versioned_docs/version-0.14.1/faq_querying_tables.md b/website/versioned_docs/version-0.14.1/faq_querying_tables.md
index 2152bde290ff..122037503c85 100644
--- a/website/versioned_docs/version-0.14.1/faq_querying_tables.md
+++ b/website/versioned_docs/version-0.14.1/faq_querying_tables.md
@@ -24,7 +24,7 @@ set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
### Does Hudi guarantee consistent reads? How to think about read optimized queries?
-Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
+Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types.md#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
Hudiβs read-optimized query is targeted for the MOR table only, with guidance around how compaction should be run to achieve predictable results. In the MOR table, the compaction, which runs every few commits (or βdeltacommitβ to be exact for the MOR table) by default, merges the base (parquet) file and corresponding change log files to a new base file within each file group, so that the snapshot query serving the latest data immediately after compaction reads the base files only.Β Simil [...]
diff --git a/website/versioned_docs/version-0.14.1/faq_storage.md b/website/versioned_docs/version-0.14.1/faq_storage.md
index 5b554b738098..810b39e569f1 100644
--- a/website/versioned_docs/version-0.14.1/faq_storage.md
+++ b/website/versioned_docs/version-0.14.1/faq_storage.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### Does Hudi support cloud storage/object stores?
-Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
+Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud.md) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations.md#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
### What is the difference between copy-on-write (COW) vs merge-on-read (MOR) table types?
@@ -14,11 +14,11 @@ Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop
**Merge On Read** - This storage type enables clients to ingest data quickly onto row based data format such as avro. Any new data that is written to the Hudi table using MOR table type, will write new log/delta files that internally store the data as avro encoded bytes. A compaction process (configured as inline or asynchronous) will convert log file format to columnar file format (parquet). Two different InputFormats expose 2 different views of this data, Read Optimized view exposes co [...]
-More details can be found [here](concepts/) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
+More details can be found [here](concepts.md) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
### How do I migrate my data to Hudi?
-Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide/), including ways to doing partial migrations.
+Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables.md#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide.md), including ways to doing partial migrations.
### How to convert an existing COW table to MOR?
@@ -34,7 +34,7 @@ But manually changing it will result in checksum errors. So, we have to go via h
### How can I find the average record size in a commit?
-The `commit showpartitons` command in [HUDI CLI](cli) will show both "bytes written" and
+The `commit showpartitons` command in [HUDI CLI](cli.md) will show both "bytes written" and
"records inserted." Divide the bytes written by records inserted to find the average size. Note that this answer assumes
@@ -91,11 +91,11 @@ spark.read.parquet("your_data_set/path/to/month").limit(n) // Limit n records
.save(basePath);
```
-For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli).
+For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli.md).
### Why does maintain record level commit metadata? Isn't tracking table version at file level good enough?Β
-By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hudi [...]
+By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering.md#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hu [...]
### Why partition fields are also stored in parquet files in addition to the partition path ?
diff --git a/website/versioned_docs/version-0.14.1/faq_table_services.md b/website/versioned_docs/version-0.14.1/faq_table_services.md
index a3fb3674d1a3..2ddcc1bd98e3 100644
--- a/website/versioned_docs/version-0.14.1/faq_table_services.md
+++ b/website/versioned_docs/version-0.14.1/faq_table_services.md
@@ -6,11 +6,11 @@ keywords: [hudi, writing, reading]
### What does the Hudi cleaner do?
-The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be read [...]
+The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations.md#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be r [...]
### How do I run compaction for a MOR table?
-Simplest way to run compaction on MOR table is to run the [compaction inline](configurations#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted most o [...]
+Simplest way to run compaction on MOR table is to run the [compaction inline](configurations.md#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted mos [...]
That said, for obvious reasons of not blocking ingesting for compaction, you may want to run it asynchronously as well. This can be done either via a separate [compaction job](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java) that is scheduled by your workflow scheduler/notebook independently. If you are using Hudi Streamer, then you can run in [continuous mode](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca [...]
@@ -40,7 +40,7 @@ Depending on how you write to Hudi these are the possible options currently.
### How to disable all table services in case of multiple writers?
-[hoodie.table.services.enabled](configurations#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
+[hoodie.table.services.enabled](configurations.md#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
### Why does Hudi retain at-least one previous commit even after setting hoodie.cleaner.commits.retained': 1 ?
diff --git a/website/versioned_docs/version-0.14.1/faq_writing_tables.md b/website/versioned_docs/version-0.14.1/faq_writing_tables.md
index 1b6dcfb6bdbe..3169d5462e2c 100644
--- a/website/versioned_docs/version-0.14.1/faq_writing_tables.md
+++ b/website/versioned_docs/version-0.14.1/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion#deltastreamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API and us [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations.md) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion.md#deltastreamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API a [...]
### How is a Hudi writer job deployed?
@@ -72,17 +72,17 @@ GDPR has made deletes a must-have tool in everyone's data management toolbox. Hu
### Should I need to worry about deleting all copies of the records in case of duplicates?
-No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https://githu [...]
+No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables.md#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables.md#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https:/ [...]
### How does Hudi handle duplicate record keys in an input?
-When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
+When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
-For an insert or bulk\_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or i [...]
+For an insert or bulk\_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) o [...]
### How can I pass hudi configurations to my spark writer job?
-Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations/). Invoking _\--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
+Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations.md). Invoking _\--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
* For Spark DataSource, you can use the "options" API of DataFrameWriter to pass in these configs.
@@ -108,28 +108,28 @@ hoodie.datasource.write.hive_style_partitioning: true
### Can I register my Hudi table with Apache Hive metastore?
-Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations#hoodiedatasourcehive_syncenable).
+Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore.md#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations.md#hoodiedatasourcehive_syncenable).
### What's Hudi's schema evolution story?
Hudi uses Avro as the internal canonical representation for records, primarily due to its nice [schema compatibility & evolution](https://docs.confluent.io/platform/current/schema-registry/avro.html) properties. This is a key aspect of having reliability in your ingestion or ETL pipelines. As long as the schema passed to Hudi (either explicitly in Hudi Streamer schema provider configs or implicitly by Spark Datasource's Dataset schemas) is backwards compatible (e.g no field deletes, only [...]
-Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution) for more details on Schema-on-read for Spark..
+Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution.md) for more details on Schema-on-read for Spark..
### What performance/ingest latency can I expect for Hudi writing?
-The speed at which you can write into Hudi depends on the [write operation](write_operations) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when try [...]
+The speed at which you can write into Hudi depends on the [write operation](write_operations.md) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when [...]
| Storage Type | Type of workload | Performance | Tips |
| ---| ---| ---| --- |
-| copy on write | bulk\_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism) to get right number of files. use insert if you want this auto tuned . Configure [hoodie.bulkinsert.sort.mode](configurations#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value NONE offers the fastest performance and matches `spark.write.parquet()` [...]
-| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
-| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or \<50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For e.g, if a partition has 1000 f [...]
+| copy on write | bulk\_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism) to get right number of files. use insert if you want this auto tuned . Configure [hoodie.bulkinsert.sort.mode](configurations.md#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value NONE offers the fastest performance and matches `spark.write.parqu [...]
+| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations.md#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
+| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or \<50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For e.g, if a partition has 1000 f [...]
| merge on read | bulk insert | Currently new data only goes to parquet files and thus performance here should be similar to copy\_on\_write bulk insert. This has the nice side-effect of getting data into parquet directly for query performance. [This GitHub issue](https://github.com/apache/hudi/issues/14468) will add support for logging inserts directly and this up drastically. | |
| merge on read | insert | Similar to above | |
| merge on read | upsert/ de-duplicate & insert | Indexing performance would remain the same as copy-on-write, while ingest latency for updates (costliest I/O operation in copy\_on\_write) are sent to log files and thus with asynchronous compaction provides very very good ingest performance with low write amplification. | |
-Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide) for more tips on JVM and other configurations.
+Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide.md) for more tips on JVM and other configurations.
### What performance can I expect for Hudi reading/queries?
@@ -147,11 +147,11 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk\_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations#hoodieparquetmaxfilesize) and the [soft limit](configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on previou [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations.md#hoodieparquetmaxfilesize) and the [soft limit](configurations.md#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on p [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
-* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations#hoodielogfilemaxsize) and a [factor](configurations#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
+* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations.md#hoodielogfilemaxsize) and a [factor](configurations.md#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
* Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations as above for the COPY\_ON\_WRITE case applies.
NOTE : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the compaction has been p [...]
@@ -173,7 +173,7 @@ hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPa
With each commit, Hudi creates a new table version in the metastore. This can be reduced by setting the option
-[hoodie.datasource.meta\_sync.condition.sync](configurations#hoodiedatasourcemeta_syncconditionsync) to true.
+[hoodie.datasource.meta\_sync.condition.sync](configurations.md#hoodiedatasourcemeta_syncconditionsync) to true.
This will ensure that hive sync is triggered on schema or partitions changes.
@@ -183,11 +183,11 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
+Hudi employs [optimistic concurrency control](concurrency_control.md) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/ [...]
### Can single-writer inserts have duplicates?
-By default, Hudi turns off key based de-duplication for INSERT/BULK\_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in delt [...]
+By default, Hudi turns off key based de-duplication for INSERT/BULK\_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in d [...]
### Can concurrent inserts cause duplicates?
diff --git a/website/versioned_docs/version-0.14.1/file_layouts.md b/website/versioned_docs/version-0.14.1/file_layouts.md
index b8b5ca7c342a..036f07f41383 100644
--- a/website/versioned_docs/version-0.14.1/file_layouts.md
+++ b/website/versioned_docs/version-0.14.1/file_layouts.md
@@ -10,8 +10,8 @@ The following describes the general file layout structure for Apache Hudi. Pleas
* Each file group contains several file slices
* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
-Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
-file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner.md) action gets rid of unused/older file slices to reclaim space on the file system.
diff --git a/website/versioned_docs/version-0.14.1/file_sizing.md b/website/versioned_docs/version-0.14.1/file_sizing.md
index a451b09b6c58..61356932afbb 100644
--- a/website/versioned_docs/version-0.14.1/file_sizing.md
+++ b/website/versioned_docs/version-0.14.1/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner.md)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md
index 6cf38b41d9c9..b56a3d6add9e 100644
--- a/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md
@@ -58,7 +58,7 @@ quick start tool for SQL users.
### Start Flink SQL client
Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
-You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore#install)), or download it from the
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
Now start the SQL CLI:
@@ -280,7 +280,7 @@ Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-e
>
This statement queries snapshot view of the dataset.
-Refers to [Table types and queries](concepts#table-types--queries) for more info on all table types and query types supported.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
## Update Data
@@ -323,7 +323,7 @@ Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/ma
</Tabs
>
-[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts)
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
denoted by the timestamp.
@@ -404,7 +404,7 @@ select * from t1;
:::info Key requirements
The bundle jar with **hive profile** is needed for streaming query, by default the officially released flink bundle is built **without**
-**hive profile**, the jar needs to be built manually, see [Build Flink Bundle Jar](syncing_metastore#install) for more details.
+**hive profile**, the jar needs to be built manually, see [Build Flink Bundle Jar](syncing_metastore.md#install) for more details.
:::
## Change Data Capture Query
@@ -453,19 +453,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](hoodie_streaming_ingestion#cdc-ingestion), [Bulk Insert](hoodie_streaming_ingestion#bulk-insert), [Index Bootstrap](hoodie_streaming_ingestion#index-bootstrap), [Changelog Mode](hoodie_streaming_ingestion#changelog-mode) and [Append Mode](hoodie_streaming_ingestion#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](writing_data#non-blocking-concurrenc [...]
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](querying_data#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
-- **Optimization**: Offline compaction is supported [Offline Compaction](compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](hoodie_streaming_ingestion.md#cdc-ingestion), [Bulk Insert](hoodie_streaming_ingestion.md#bulk-insert), [Index Bootstrap](hoodie_streaming_ingestion.md#index-bootstrap), [Changelog Mode](hoodie_streaming_ingestion.md#changelog-mode) and [Append Mode](hoodie_streaming_ingestion.md#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](writing_data.md#non-b [...]
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](querying_data.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
- - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- - [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+ - [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -474,5 +474,5 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it. Also, if you are looking for ways to migrate your existing data
-to Hudi, refer to [migration guide](migration_guide).
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-0.14.1/hoodie_cleaner.md b/website/versioned_docs/version-0.14.1/hoodie_cleaner.md
index fa4c43ec7688..13d2270c70df 100644
--- a/website/versioned_docs/version-0.14.1/hoodie_cleaner.md
+++ b/website/versioned_docs/version-0.14.1/hoodie_cleaner.md
@@ -126,7 +126,7 @@ spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls
Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
#### CLI
-You can also use [Hudi CLI](cli) to run Hoodie Cleaner.
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
CLI provides the below commands for cleaner service:
- `cleans show`
diff --git a/website/versioned_docs/version-0.14.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-0.14.1/hoodie_streaming_ingestion.md
index 809c549a55ba..c3f824ef712d 100644
--- a/website/versioned_docs/version-0.14.1/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-0.14.1/hoodie_streaming_ingestion.md
@@ -171,7 +171,7 @@ and then ingest it as follows.
--op BULK_INSERT
```
-In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide).
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
### Note on hudi utilities bundle usage for different spark versions
From 0.11.0 release, we start to provide a new `hudi-utilities-slim-bundle` which aims to exclude dependencies that can
@@ -237,7 +237,7 @@ A Hudi Streamer job can then be triggered as follows:
--op BULK_INSERT
```
-Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control) section
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
## Checkpointing
`HoodieStreamer` uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data.
@@ -603,7 +603,7 @@ frequent `file handle` switching.
:::note
The parallelism of `bulk_insert` is specified by `write.tasks`. The parallelism will affect the number of small files.
In theory, the parallelism of `bulk_insert` is the number of `bucket`s (In particular, when each bucket writes to maximum file size, it
-will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks).
+will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks).
:::
#### Options
@@ -611,7 +611,7 @@ will rollover to the new file handle. Finally, `the number of files` >= [`write.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.operation` | `true` | `upsert` | Setting as `bulk_insert` to open this function |
-| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks) |
+| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks) |
| `write.bulk_insert.shuffle_input` | `false` | `true` | Whether to shuffle data according to the input field before writing. Enabling this option will reduce the number of small files, but there may be a risk of data skew |
| `write.bulk_insert.sort_input` | `false` | `true` | Whether to sort data according to the input field before writing. Enabling this option will reduce the number of small files when a write task writes multiple partitions |
| `write.sort.memory` | `false` | `128` | Available managed memory of sort operator. default `128` MB |
diff --git a/website/versioned_docs/version-0.14.1/indexing.md b/website/versioned_docs/version-0.14.1/indexing.md
index 53e883c38561..6dcbf454e6f1 100644
--- a/website/versioned_docs/version-0.14.1/indexing.md
+++ b/website/versioned_docs/version-0.14.1/indexing.md
@@ -11,9 +11,9 @@ Hudi provides efficient upserts, by mapping a given hoodie key (record key + par
This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
mapped file group contains all versions of a group of records.
-For [Copy-On-Write tables](table_types#copy-on-write-table), this enables fast upsert/delete operations, by
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), this enables fast upsert/delete operations, by
avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](table_types#merge-on-read-table), this design allows Hudi to bound the amount of
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), this design allows Hudi to bound the amount of
records any given base file needs to be merged against.
Specifically, a given base file needs to merged only against updates for records that are part of that base file. In contrast,
designs without an indexing component (e.g: [Apache Hive ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)),
diff --git a/website/versioned_docs/version-0.14.1/metadata.md b/website/versioned_docs/version-0.14.1/metadata.md
index ea80418658af..bc37d0fcd7e8 100644
--- a/website/versioned_docs/version-0.14.1/metadata.md
+++ b/website/versioned_docs/version-0.14.1/metadata.md
@@ -106,7 +106,7 @@ Following are the Spark based basic configs that are needed to enable metadata a
The metadata table with synchronous updates and metadata-table-based file listing are enabled by default.
There are prerequisite configurations and steps in [Deployment considerations](#deployment-considerations-for-metadata-table) to
safely use this feature. The metadata table and related file listing functionality can still be turned off by setting
-[`hoodie.metadata.enable`](configurations#hoodiemetadataenable) to `false`. The
+[`hoodie.metadata.enable`](configurations.md#hoodiemetadataenable) to `false`. The
[multi-modal index](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi) are
disabled by default and can be enabled in write side explicitly using the above configs.
@@ -151,17 +151,17 @@ corresponding configs across Spark and Flink readers.
## Deployment considerations for metadata Table
To ensure that metadata table stays up to date, all write operations on the same Hudi table need additional configurations
besides the above in different deployment models. Before enabling metadata table, all writers on the same table must
-be stopped. Please refer to the different [deployment models](concurrency_control#deployment-models-with-supported-concurrency-controls)
+be stopped. Please refer to the different [deployment models](concurrency_control.md#deployment-models-with-supported-concurrency-controls)
for more details on each model. This section only highlights how to safely enable metadata table in different deployment models.
### Deployment Model A: Single writer with inline table services
-In [Model A](concurrency_control#model-a-single-writer-with-inline-table-services), after setting [`hoodie.metadata.enable`](configurations#hoodiemetadataenable) to `true`, restarting
+In [Model A](concurrency_control.md#model-a-single-writer-with-inline-table-services), after setting [`hoodie.metadata.enable`](configurations.md#hoodiemetadataenable) to `true`, restarting
the single writer is sufficient to safely enable metadata table.
### Deployment Model B: Single writer with async table services
-If your current deployment model is [Model B](concurrency_control#model-b-single-writer-with-async-table-services), enabling metadata
+If your current deployment model is [Model B](concurrency_control.md#model-b-single-writer-with-async-table-services), enabling metadata
table requires adding optimistic concurrency control along with suggested lock provider like below.
```properties
hoodie.write.concurrency.mode=optimistic_concurrency_control
@@ -179,7 +179,7 @@ process which cannot rely on the in-process lock provider.
### Deployment Model C: Multi-writer
-If your current deployment model is [multi-writer](concurrency_control#model-c-multi-writer) along with a lock
+If your current deployment model is [multi-writer](concurrency_control.md#model-c-multi-writer) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only partial writers leads to loss of data from the inconsistent writer. So, ensure you enable
@@ -190,7 +190,7 @@ hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.write.lock.provider=<distributed-lock-provider-classname>
```
-Note that there are different external [lock providers available](concurrency_control#external-locking-and-lock-providers)
+Note that there are different external [lock providers available](concurrency_control.md#external-locking-and-lock-providers)
to choose from.
## Related Resources
diff --git a/website/versioned_docs/version-0.14.1/metadata_indexing.md b/website/versioned_docs/version-0.14.1/metadata_indexing.md
index 54a446a0b7ed..b96b53025456 100644
--- a/website/versioned_docs/version-0.14.1/metadata_indexing.md
+++ b/website/versioned_docs/version-0.14.1/metadata_indexing.md
@@ -5,7 +5,7 @@ toc: true
last_modified_at:
---
-Hudi maintains a scalable [metadata](metadata) that has some auxiliary data about the table.
+Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data about the table.
The [pluggable indexing subsystem](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi)
of Hudi depends on the metadata table. Different types of index, from `files` index for locating records efficiently
to `column_stats` index for data skipping, are part of the metadata table. A fundamental tradeoff in any data system
@@ -25,7 +25,7 @@ feature, please check out [this blog](https://www.onehouse.ai/blog/asynchronous-
## Setup Async Indexing
-First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) which will continuously write data
+First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) which will continuously write data
from raw parquet to Hudi table. We used the widely available [NY Taxi dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/), whose setup details are as below:
<details>
<summary>Ingestion write config</summary>
@@ -78,7 +78,7 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](metadata#metadata-table-indices) along with [configs](metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
+metadata table can be explored [here](metadata.md#metadata-table-indices) along with [configs](metadata.md#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
diff --git a/website/versioned_docs/version-0.14.1/metrics.md b/website/versioned_docs/version-0.14.1/metrics.md
index c6f2833f5ada..c389689716be 100644
--- a/website/versioned_docs/version-0.14.1/metrics.md
+++ b/website/versioned_docs/version-0.14.1/metrics.md
@@ -6,7 +6,7 @@ toc: true
last_modified_at: 2020-06-20T15:59:57-04:00
---
-In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations#METRICS).
+In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations.md#METRICS).
## MetricsReporter
@@ -17,7 +17,7 @@ MetricsReporter provides APIs for reporting `HoodieMetrics` to user-specified ba
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
#### Configurations
-The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Jmx).
+The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Jmx).
```properties
hoodie.metrics.on=true
@@ -37,7 +37,7 @@ As configured above, JmxMetricsReporter will started JMX server on port 4001. We
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send `HoodieMetrics` to it.
#### Configurations
-The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations#Metrics-Configurations-for-Graphite).
+The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Graphite).
```properties
hoodie.metrics.on=true
@@ -58,7 +58,7 @@ DatadogMetricsReporter is an implementation of Datadog reporter.
A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
#### Configurations
-The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Datadog-reporter).
+The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Datadog-reporter).
```properties
hoodie.metrics.on=true
@@ -138,7 +138,7 @@ tuned are in the `HoodieMetricsCloudWatchConfig` class.
Allows users to define a custom metrics reporter.
#### Configurations
-The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations).
+The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations).
```properties
hoodie.metrics.on=true
diff --git a/website/versioned_docs/version-0.14.1/migration_guide.md b/website/versioned_docs/version-0.14.1/migration_guide.md
index 2baf7ee0dc0f..b63bc3fbd596 100644
--- a/website/versioned_docs/version-0.14.1/migration_guide.md
+++ b/website/versioned_docs/version-0.14.1/migration_guide.md
@@ -40,7 +40,7 @@ There are a few options when choosing this approach.
#### Using Hudi Streamer
-Use the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool. HoodieStreamer supports bootstrap with
+Use the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool. HoodieStreamer supports bootstrap with
--run-bootstrap command line option. There are two types of bootstrap, METADATA_ONLY and FULL_RECORD. METADATA_ONLY will
generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will
perform a full copy/rewrite of the data as a Hudi table. Additionally, once can choose selective partitions using regex
@@ -88,7 +88,7 @@ Refer to [Bootstrap procedure](https://hudi.apache.org/docs/next/procedures#boot
#### Using Hudi CLI
Write your own custom logic of how to load an existing table into a Hudi managed one. Please read about the RDD API
-[here](quick-start-guide). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
+[here](quick-start-guide.md). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hudi-cli && ./hudi-cli.sh`.
```java
diff --git a/website/versioned_docs/version-0.14.1/overview.mdx b/website/versioned_docs/version-0.14.1/overview.mdx
index 6099f9b43a36..8f8a8ffd544d 100644
--- a/website/versioned_docs/version-0.14.1/overview.mdx
+++ b/website/versioned_docs/version-0.14.1/overview.mdx
@@ -13,27 +13,27 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced βhoodieβ) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
-Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](sql_ddl),
-[transactions](timeline), [efficient upserts/deletes](write_operations), [advanced indexes](indexing),
-[streaming ingestion services](hoodie_streaming_ingestion), data [clustering](clustering)/[compaction](compaction) optimizations,
-and [concurrency](concurrency_control) all while keeping your data in open source file formats.
+Apache Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](sql_ddl.md),
+[transactions](timeline.md), [efficient upserts/deletes](write_operations.md), [advanced indexes](indexing.md),
+[streaming ingestion services](hoodie_streaming_ingestion.md), data [clustering](clustering.md)/[compaction](compaction.md) optimizations,
+and [concurrency](concurrency_control.md) all while keeping your data in open source file formats.
Not only is Apache Hudi great for streaming workloads, but it also allows you to create efficient incremental batch pipelines.
-Read the docs for more [use case descriptions](use_cases) and check out [who's using Hudi](/powered-by), to see how some of the
+Read the docs for more [use case descriptions](use_cases.md) and check out [who's using Hudi](/powered-by), to see how some of the
largest data lakes in the world including [Uber](https://www.uber.com/en-IN/blog/uber-big-data-platform/), [Amazon](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/),
[ByteDance](http://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance),
[Robinhood](https://s.apache.org/hudi-robinhood-talk) and more are transforming their production data lakes with Hudi.
-Apache Hudi can easily be used on any [cloud storage platform](cloud).
+Apache Hudi can easily be used on any [cloud storage platform](cloud.md).
Hudiβs advanced performance optimizations, make analytical workloads faster with any of
the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
-- [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
-- [Hudi File Layout](file_layouts) - How the files are laid out on storage
-- [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
-- [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+- [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+- [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+- [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -41,11 +41,11 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
## Getting Started
Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:
-- [Spark Quick Start Guide](quick-start-guide) β if you primarily use Apache Spark
-- [Flink Quick Start Guide](flink-quick-start-guide) β if you primarily use Apache Flink
+- [Spark Quick Start Guide](quick-start-guide.md) β if you primarily use Apache Spark
+- [Flink Quick Start Guide](flink-quick-start-guide.md) β if you primarily use Apache Flink
If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the Docker Demo:
-- [Docker Demo](docker_demo)
+- [Docker Demo](docker_demo.md)
## Connect With The Community
Apache Hudi is community focused and community led and welcomes new-comers with open arms. Leverage the following
diff --git a/website/versioned_docs/version-0.14.1/performance.md b/website/versioned_docs/version-0.14.1/performance.md
index cd9de72d5a08..9fec829e492f 100644
--- a/website/versioned_docs/version-0.14.1/performance.md
+++ b/website/versioned_docs/version-0.14.1/performance.md
@@ -16,12 +16,12 @@ Here are some ways to efficiently manage the storage of your Hudi tables.
- The [small file handling feature](configurations/#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](configurations#hoodieparquetcompressionratio),
+- Cleaner can be [configured](configurations.md#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](configurations.md#hoodieparquetmaxfilesize), [log files](configurations.md#hoodielogfilemaxsize) & expected [compression ratio](configurations.md#hoodieparquetcompressionratio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
-- Intelligently tuning the [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
+- Intelligently tuning the [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be changed without re-clustering the table. Writes will simply expand given file groups with new updates/inserts as explained before.
-- For workloads with heavy updates, the [merge-on-read table](concepts#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
+- For workloads with heavy updates, the [merge-on-read table](concepts.md#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
## Performance Optimizations
@@ -119,11 +119,11 @@ For very large tables (1Tb+, 10s of 1000s of files), Data skipping could
To unlock the power of Data Skipping you will need to
-1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
+1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing.md)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
2. Enable Data Skipping in your queries, using `hoodie.metadata.index.column.stats.enable=true` (to enable Column Stats Index being populated on the write path, disabled by default)
:::note
-If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
+If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing.md) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
:::
To enable Data Skipping in your queries make sure to set following properties to `true` (on the read path):
diff --git a/website/versioned_docs/version-0.14.1/precommit_validator.md b/website/versioned_docs/version-0.14.1/precommit_validator.md
index 2ecd31665891..440eee9ac1ef 100644
--- a/website/versioned_docs/version-0.14.1/precommit_validator.md
+++ b/website/versioned_docs/version-0.14.1/precommit_validator.md
@@ -8,7 +8,7 @@ Data quality refers to the overall accuracy, completeness, consistency, and vali
Hudi offers **Pre-Commit Validators** that allow you to ensure that your data meets certain data quality expectations as you are writing with Hudi Streamer or Spark Datasource writers.
:::note
-Pre-commit validators are skipped when using the [BULK_INSERT](write_operations#bulk_insert) write operation type.
+Pre-commit validators are skipped when using the [BULK_INSERT](write_operations.md#bulk_insert) write operation type.
:::
Multiple class names can be separated by `,` delimiter.
diff --git a/website/versioned_docs/version-0.14.1/procedures.md b/website/versioned_docs/version-0.14.1/procedures.md
index 0a895560df09..2c106dcfc956 100644
--- a/website/versioned_docs/version-0.14.1/procedures.md
+++ b/website/versioned_docs/version-0.14.1/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations.md#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations.md#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations.md#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations.md#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1533,11 +1533,11 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
-| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
-| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [clean_policy](configurations.md#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum qu [...]
+| [retain_commits](configurations.md#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations.md#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of [...]
+| [file_versions_retained](configurations.md#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations.md#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
@@ -1633,12 +1633,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations.md#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations.md#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations.md#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations.md#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations.md#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations.md#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1848,18 +1848,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations.md#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations.md#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations.md#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
+| [parallelism](configurations.md#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-0.14.1/querying_data.md b/website/versioned_docs/version-0.14.1/querying_data.md
index b0102fb2b335..4cc1d3921034 100644
--- a/website/versioned_docs/version-0.14.1/querying_data.md
+++ b/website/versioned_docs/version-0.14.1/querying_data.md
@@ -7,10 +7,10 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl.md), [SQL DML](sql_dml.md), [SQL Queries](sql_queries.md) and [Procedures](procedures.md) for the latest documentation.
:::
-Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts#query-types).
+Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts.md#query-types).
Once the table is synced to the Hive metastore, it provides external Hive tables backed by Hudi's custom inputformats. Once the proper hudi
bundle has been installed, the table can be queried by popular query engines like Hive, Spark SQL, Flink, Trino and PrestoDB.
@@ -19,7 +19,7 @@ In sections, below we will discuss specific setup to access different query type
## Spark Datasource
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
-See the [Spark Quick Start](quick-start-guide) for more examples of Spark datasource reading queries.
+See the [Spark Quick Start](quick-start-guide.md) for more examples of Spark datasource reading queries.
**Setup**
@@ -56,8 +56,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Incremental Queries](quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](configurations#SPARK_DATASOURCE) section, to view all datasource options.
+For examples, refer to [Incremental Queries](quick-start-guide.md#incremental-query) in the Spark quickstart.
+Please refer to [configurations](configurations.md#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.14.1/quick-start-guide.md b/website/versioned_docs/version-0.14.1/quick-start-guide.md
index 92aa1a132907..b7a14218e38d 100644
--- a/website/versioned_docs/version-0.14.1/quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.1/quick-start-guide.md
@@ -262,7 +262,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl.md) reference guide.
</TabItem>
@@ -306,7 +306,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -339,7 +339,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -348,7 +348,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml.md#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -460,7 +460,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml.md#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -490,7 +490,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts.md).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -578,7 +578,7 @@ MERGE statement either using `SET *` or using `SET column1 = expression1 [, colu
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data.md#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -948,7 +948,7 @@ Note that CDC queries are currently only supported on Copy-on-Write tables.
The examples thus far have showcased one of the two table types, that Hudi supports - Copy-on-Write (COW) tables. Hudi also supports
a more advanced write-optimized table type called Merge-on-Read (MOR) tables, that can balance read and write performance in a more
-flexible manner. See [table types](table_types) for more details.
+flexible manner. See [table types](table_types.md) for more details.
Any of these examples can be run on a Merge-on-Read table by simply changing the table type to MOR, while creating the table, as below.
@@ -1074,7 +1074,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations.md).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1147,29 +1147,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide.md).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
-For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl.md) and [Spark SQL DML](sql_dml.md) reference guides.
+For alter table commands, check out [this](sql_ddl.md#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures.md) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion.md#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](hoodie_streaming_ingestion#structured-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](hoodie_streaming_ingestion.md#structured-streaming) for more.
-Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](writing_data).
+Check out more information on [modeling data in Hudi](faq_general.md#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](writing_data.md).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it.
+steps [here](docker_demo.md) to get a taste for it.
diff --git a/website/versioned_docs/version-0.14.1/record_payload.md b/website/versioned_docs/version-0.14.1/record_payload.md
index 50f6f6ec57a5..49848a0d41f9 100644
--- a/website/versioned_docs/version-0.14.1/record_payload.md
+++ b/website/versioned_docs/version-0.14.1/record_payload.md
@@ -105,7 +105,7 @@ hoodie.datasource.write.payload.class=org.apache.hudi.common.model.EventTimeAvro
```
This is the default record payload for Flink based writing. Some use cases require merging records by event time and
thus event time plays the role of an ordering field. This payload is particularly useful in the case of late-arriving data.
-For such use cases, users need to set the [payload event time field](configurations#RECORD_PAYLOAD) configuration.
+For such use cases, users need to set the [payload event time field](configurations.md#RECORD_PAYLOAD) configuration.
#### OverwriteNonDefaultsWithLatestAvroPayload
```scala
@@ -171,7 +171,7 @@ provides support for seamlessly applying changes captured via Debezium for MySQL
provides support for applying changes captured via Amazon Database Migration Service onto S3.
Record payloads are tunable to suit many use cases. Please check out the configurations
-listed [here](configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
+listed [here](configurations.md#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.14.1/s3_hoodie.md b/website/versioned_docs/version-0.14.1/s3_hoodie.md
index 5faad6e62be9..fe3c9e513cfe 100644
--- a/website/versioned_docs/version-0.14.1/s3_hoodie.md
+++ b/website/versioned_docs/version-0.14.1/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](hoodie_cleaner.md) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.14.1/sql_ddl.md b/website/versioned_docs/version-0.14.1/sql_ddl.md
index 684a98dd0f2f..a5865f50f6e2 100644
--- a/website/versioned_docs/version-0.14.1/sql_ddl.md
+++ b/website/versioned_docs/version-0.14.1/sql_ddl.md
@@ -75,7 +75,7 @@ should be specified as `PARTITIONED BY (dt, hh)`.
### Create table with record keys and ordering fields
-As discussed [here](quick-start-guide#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
Typically, this is also accompanied by configuring a `preCombineField` option to deal with out-of-order data and potential
duplicate records with the same key in the incoming writes.
@@ -104,7 +104,7 @@ TBLPROPERTIES (
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion.md#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -378,7 +378,7 @@ Users can set table properties while creating a table. The important table prope
| Parameter Name | Default | Description |
|------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types) |
+| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types.md) |
| primaryKey | uuid | The primary key field names of the table separated by commas. Same as `hoodie.datasource.write.recordkey.field`. If this config is ignored, hudi will auto-generate primary keys. If explicitly set, primary key generation will honor user configuration. |
| preCombineField | | The pre-combine field of the table. It is used for resolving the final version of the record among multiple versions. Generally, `event time` or another similar column will be used for ordering purposes. Hudi will be able to handle out-of-order data using the preCombine field value. |
@@ -389,7 +389,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control.md#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -612,7 +612,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations.md#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-0.14.1/sql_dml.md b/website/versioned_docs/version-0.14.1/sql_dml.md
index 47c82fcfea28..9ca9cd9fc471 100644
--- a/website/versioned_docs/version-0.14.1/sql_dml.md
+++ b/website/versioned_docs/version-0.14.1/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
### Insert Into
@@ -25,7 +25,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -45,7 +45,7 @@ INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000
```
:::note Mapping to write operations
-Hudi offers flexibility in choosing the underlying [write operation](write_operations) of a `INSERT INTO` statement using
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
and *"upsert"* (with deduplication/merging). If a precombine field is not set, *"insert"* is chosen as the default. For a table with preCombine field set,
*"upsert"* is chosen as the default operation.
diff --git a/website/versioned_docs/version-0.14.1/sql_queries.md b/website/versioned_docs/version-0.14.1/sql_queries.md
index 8276af0e0e6d..6df2c307d629 100644
--- a/website/versioned_docs/version-0.14.1/sql_queries.md
+++ b/website/versioned_docs/version-0.14.1/sql_queries.md
@@ -8,11 +8,11 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-Hudi stores and organizes data on storage while providing different ways of [querying](concepts#query-types), across a wide range of query engines.
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
This page will show how to issue different queries and discuss any specific instructions for each query engine.
## Spark SQL
-The Spark [quickstart](quick-start-guide) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
@@ -91,11 +91,11 @@ For e.g the table has received 10 million modifications across 1 million records
see all changes in a given time window and not just the latest values.
:::
-Please refer to [configurations](basic_configurations) section for the important configuration options.
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
## Flink SQL
Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
-relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide).
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-0.14.1/structure.md b/website/versioned_docs/version-0.14.1/structure.md
index 31281b68f71c..9f78f89f16f5 100644
--- a/website/versioned_docs/version-0.14.1/structure.md
+++ b/website/versioned_docs/version-0.14.1/structure.md
@@ -17,4 +17,4 @@ Hudi (pronounced βHoodieβ) ingests & manages storage of large analytical tab
By carefully managing how data is laid out in storage & how itβs exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [PrestoDB](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) [...]
-Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide) for a demo.
+Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide.md) for a demo.
diff --git a/website/versioned_docs/version-0.14.1/syncing_aws_glue_data_catalog.md b/website/versioned_docs/version-0.14.1/syncing_aws_glue_data_catalog.md
index e54c6d528876..b625c36ef3f5 100644
--- a/website/versioned_docs/version-0.14.1/syncing_aws_glue_data_catalog.md
+++ b/website/versioned_docs/version-0.14.1/syncing_aws_glue_data_catalog.md
@@ -10,7 +10,7 @@ and send them to AWS Glue.
### Configurations
There is no additional configuration for using `AwsGlueCatalogSyncTool`; you just need to set it as one of the sync tool
-classes for `HoodieStreamer` and everything configured as shown in [Sync to Hive Metastore](syncing_metastore) will
+classes for `HoodieStreamer` and everything configured as shown in [Sync to Hive Metastore](syncing_metastore.md) will
be passed along.
```shell
diff --git a/website/versioned_docs/version-0.14.1/syncing_metastore.md b/website/versioned_docs/version-0.14.1/syncing_metastore.md
index ed427195f368..4417cd3d9124 100644
--- a/website/versioned_docs/version-0.14.1/syncing_metastore.md
+++ b/website/versioned_docs/version-0.14.1/syncing_metastore.md
@@ -12,7 +12,7 @@ table to Hive metastore.
## Hive Sync Tool
-Writing data with [DataSource](writing_data) writer or [HoodieStreamer](hoodie_streaming_ingestion) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
+Writing data with [DataSource](writing_data.md) writer or [HoodieStreamer](hoodie_streaming_ingestion.md) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
In case, it's preferable to run this from commandline or in an independent jvm, Hudi provides a `HiveSyncTool`, which can be invoked as below,
once you have built the hudi-hive module. Following is how we sync the above Datasource Writer written table to Hive metastore.
diff --git a/website/versioned_docs/version-0.14.1/use_cases.md b/website/versioned_docs/version-0.14.1/use_cases.md
index fb6061b8d1b5..61ea8ccddb8a 100644
--- a/website/versioned_docs/version-0.14.1/use_cases.md
+++ b/website/versioned_docs/version-0.14.1/use_cases.md
@@ -85,7 +85,7 @@ stream processing world to ensure pipelines don't break from non backwards compa
### ACID Transactions
Along with a table, Apache Hudi brings ACID transactional guarantees to a data lake.
-Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline), stamped with an
+Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline.md), stamped with an
instant time that denotes the time at which the action
is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between writer processes
(that issue userβs upserts/deletes), table services (that write data/metadata to optimize/perform bookkeeping) and readers
@@ -127,12 +127,12 @@ cost savings for your data lake.
Some examples of the Apache Hudi services that make this performance optimization easy include:
-- [Auto File Sizing](file_sizing) - to solve the "small files" problem.
-- [Clustering](clustering) - to co-locate data next to each other.
-- [Compaction](compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](indexing) - for efficient upserts and deletes.
+- [Auto File Sizing](file_sizing.md) - to solve the "small files" problem.
+- [Clustering](clustering.md) - to co-locate data next to each other.
+- [Compaction](compaction.md) - to allow tuning of low latency ingestion and fast read queries.
+- [Indexing](indexing.md) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner.md) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.14.1/write_operations.md b/website/versioned_docs/version-0.14.1/write_operations.md
index 9b9a862bd0b9..ac2a107b27cb 100644
--- a/website/versioned_docs/version-0.14.1/write_operations.md
+++ b/website/versioned_docs/version-0.14.1/write_operations.md
@@ -6,7 +6,7 @@ last_modified_at:
---
It may be helpful to understand the different write operations of Hudi and how best to leverage them. These operations
-can be chosen/changed across each commit/deltacommit issued against the table. See the [How To docs on Writing Data](writing_data)
+can be chosen/changed across each commit/deltacommit issued against the table. See the [How To docs on Writing Data](writing_data.md)
to see more examples.
## Operation Types
@@ -86,27 +86,27 @@ Here are the basic configs relevant to the write operations types mentioned abov
## Writing path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](configurations#hoodiecombinebeforeinsert)
+1. [Deduping](configurations.md#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](indexing)
+2. [Index Lookup](indexing.md)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](file_sizing)
+3. [File Sizing](file_sizing.md)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](file_layouts)
+4. [Partitioning](file_layouts.md)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](indexing)
+6. Update [Index](indexing.md)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
-8. [Clean](hoodie_cleaner) (if needed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data.md#commit-notifications) is exposed)
+8. [Clean](hoodie_cleaner.md) (if needed)
1. Following the commit, cleaning is invoked if needed.
-9. [Compaction](compaction)
+9. [Compaction](compaction.md)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
10. Archive
- 1. Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
+ 1. Lastly, we perform an archival step which moves old [timeline](timeline.md) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-0.14.1/writing_data.md b/website/versioned_docs/version-0.14.1/writing_data.md
index 18520c135b0a..27c3712c3fbb 100644
--- a/website/versioned_docs/version-0.14.1/writing_data.md
+++ b/website/versioned_docs/version-0.14.1/writing_data.md
@@ -10,14 +10,14 @@ import TabItem from '@theme/TabItem';
In this section, we will cover ways to ingest new changes from external sources or even other Hudi tables.
Currently Hudi supports following ways to write the data.
-- [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer)
+- [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer)
- [Spark Hudi Datasource](#spark-datasource-writer)
-- [Spark Structured Streaming](hoodie_streaming_ingestion#structured-streaming)
-- [Spark SQL](sql_ddl#spark-sql)
-- [Flink Writer](hoodie_streaming_ingestion#flink-ingestion)
-- [Flink SQL](sql_ddl#flink)
+- [Spark Structured Streaming](hoodie_streaming_ingestion.md#structured-streaming)
+- [Spark SQL](sql_ddl.md#spark-sql)
+- [Flink Writer](hoodie_streaming_ingestion.md#flink-ingestion)
+- [Flink SQL](sql_ddl.md#flink)
- [Java Writer](#java-writer)
-- [Kafka Connect](hoodie_streaming_ingestion#kafka-connect-sink)
+- [Kafka Connect](hoodie_streaming_ingestion.md#kafka-connect-sink)
## Spark Datasource Writer
@@ -39,15 +39,15 @@ Default value: `"partitionpath"`<br/>
**PRECOMBINE_FIELD_OPT_KEY** (Required): When two records within the same batch have the same key value, the record with the largest value from the field specified will be choosen. If you are using default payload of OverwriteWithLatestAvroPayload for HoodieRecordPayload (`WRITE_PAYLOAD_CLASS`), an incoming record will always takes precendence compared to the one in storage ignoring this `PRECOMBINE_FIELD_OPT_KEY`. <br/>
Default value: `"ts"`<br/>
-**OPERATION_OPT_KEY**: The [write operations](write_operations) to use.<br/>
+**OPERATION_OPT_KEY**: The [write operations](write_operations.md) to use.<br/>
Available values:<br/>
`UPSERT_OPERATION_OPT_VAL` (default), `BULK_INSERT_OPERATION_OPT_VAL`, `INSERT_OPERATION_OPT_VAL`, `DELETE_OPERATION_OPT_VAL`
-**TABLE_TYPE_OPT_KEY**: The [type of table](concepts#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
+**TABLE_TYPE_OPT_KEY**: The [type of table](concepts.md#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
Available values:<br/>
-[`COW_TABLE_TYPE_OPT_VAL`](concepts#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts#merge-on-read-table)
+[`COW_TABLE_TYPE_OPT_VAL`](concepts.md#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts.md#merge-on-read-table)
-**KEYGENERATOR_CLASS_OPT_KEY**: Refer to [Key Generation](key_generation) section below.
+**KEYGENERATOR_CLASS_OPT_KEY**: Refer to [Key Generation](key_generation.md) section below.
**HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY**: If using hive, specify if the table should or should not be partitioned.<br/>
Available values:<br/>
@@ -99,9 +99,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -135,9 +135,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](writing_data.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -547,7 +547,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control.md#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-0.15.0/basic_configurations.md b/website/versioned_docs/version-0.15.0/basic_configurations.md
index 518c102b69f4..ae9099b2facb 100644
--- a/website/versioned_docs/version-0.15.0/basic_configurations.md
+++ b/website/versioned_docs/version-0.15.0/basic_configurations.md
@@ -1,11 +1,11 @@
---
title: Basic Configurations
-summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
last_modified_at: 2024-06-06T12:05:06.444
---
-This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
diff --git a/website/versioned_docs/version-0.15.0/cli.md b/website/versioned_docs/version-0.15.0/cli.md
index 13fd9c0ac1ad..77bc50102bba 100644
--- a/website/versioned_docs/version-0.15.0/cli.md
+++ b/website/versioned_docs/version-0.15.0/cli.md
@@ -578,7 +578,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-0.15.0/cloud.md b/website/versioned_docs/version-0.15.0/cloud.md
index 123abd5e6bea..a3477b551bee 100644
--- a/website/versioned_docs/version-0.15.0/cloud.md
+++ b/website/versioned_docs/version-0.15.0/cloud.md
@@ -11,23 +11,23 @@ last_modified_at: 2021-10-12T10:50:00+08:00
Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
to cloud stores.
-* [AWS S3](s3_hoodie) <br/>
+* [AWS S3](s3_hoodie.md) <br/>
Configurations required for S3 and Hudi co-operability.
-* [Google Cloud Storage](gcs_hoodie) <br/>
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
Configurations required for GCS and Hudi co-operability.
-* [Alibaba Cloud OSS](oss_hoodie) <br/>
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
Configurations required for OSS and Hudi co-operability.
-* [Microsoft Azure](azure_hoodie) <br/>
+* [Microsoft Azure](azure_hoodie.md) <br/>
Configurations required for Azure and Hudi co-operability.
-* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
Configurations required for COS and Hudi co-operability.
-* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
Configurations required for IBM Cloud Object Storage and Hudi co-operability.
-* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
Configurations required for BOS and Hudi co-operability.
-* [JuiceFS](jfs_hoodie) <br/>
+* [JuiceFS](jfs_hoodie.md) <br/>
Configurations required for JuiceFS and Hudi co-operability.
-* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
Configurations required for OCI and Hudi co-operability.
:::note
diff --git a/website/versioned_docs/version-0.15.0/clustering.md b/website/versioned_docs/version-0.15.0/clustering.md
index 14787ed354a9..c5b214ab659c 100644
--- a/website/versioned_docs/version-0.15.0/clustering.md
+++ b/website/versioned_docs/version-0.15.0/clustering.md
@@ -13,7 +13,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## How is compaction different from clustering?
Hudi is modeled like a log-structured storage engine with multiple versions of the data.
-Particularly, [Merge-On-Read](table_types#merge-on-read-table)
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
@@ -93,7 +93,7 @@ update strategy.
This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
-config [hoodie.clustering.plan.strategy.class](configurations#hoodieclusteringplanstrategyclass).
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
Different plan strategies are as follows:
diff --git a/website/versioned_docs/version-0.15.0/compaction.md b/website/versioned_docs/version-0.15.0/compaction.md
index 670bd96e49f8..341b885f1e70 100644
--- a/website/versioned_docs/version-0.15.0/compaction.md
+++ b/website/versioned_docs/version-0.15.0/compaction.md
@@ -13,7 +13,7 @@ not applicable to Copy On Write(COW) tables and only applies to MOR tables.
### Why MOR tables need compaction?
To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
-data is organized in terms of [file groups](file_layouts). Each file group in a MOR table
+data is organized in terms of [file groups](file_layouts.md). Each file group in a MOR table
consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
are appended to log files.
@@ -182,7 +182,7 @@ This model may need a lock provider **if** metadata table is enabled.
:::
#### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
The compactor utility allows to do scheduling and execution of compaction.
Example:
@@ -199,7 +199,7 @@ Note, the `instant-time` parameter is now optional for the Hudi Compactor Utilit
the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
#### Hudi CLI
-Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.15.0/concurrency_control.md b/website/versioned_docs/version-0.15.0/concurrency_control.md
index 78724b99165f..6181614ea986 100644
--- a/website/versioned_docs/version-0.15.0/concurrency_control.md
+++ b/website/versioned_docs/version-0.15.0/concurrency_control.md
@@ -77,7 +77,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](sql_dml#non-blocking-concurrency-control-experimental).
+writers [here](sql_dml.md#non-blocking-concurrency-control-experimental).
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.15.0/deployment.md b/website/versioned_docs/version-0.15.0/deployment.md
index f2c385a475d1..bbff6c584415 100644
--- a/website/versioned_docs/version-0.15.0/deployment.md
+++ b/website/versioned_docs/version-0.15.0/deployment.md
@@ -26,7 +26,7 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### Hudi Streamer
-[Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
To use Hudi Streamer in Spark, the `hudi-utilities-bundle` is required, by adding
@@ -37,7 +37,7 @@ corresponding to the Spark version used, e.g.,
`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.13.0,org.apache.hudi:hudi-spark3.1-bundle_2.12:0.13.0`,
if using `hudi-utilities-bundle` solely in Spark encounters compatibility issues.
- - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Batch Writes](writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.d [...]
+As described in [Batch Writes](writing_data.md#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
@@ -237,7 +237,7 @@ hoodie.table.version=5
```
Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
-For more details on downgrade, please refer documentation [here](cli#upgrade-and-downgrade-table).
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
```shell
connect --path /tmp/hudi_table
downgrade table --toVersion 4
@@ -258,4 +258,4 @@ Currently migrating to Hudi can be done using two approaches
- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
-For more details, refer to the detailed [migration guide](migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-0.15.0/faq.md b/website/versioned_docs/version-0.15.0/faq.md
index 26c3eb50d214..c53c8ebbc68c 100644
--- a/website/versioned_docs/version-0.15.0/faq.md
+++ b/website/versioned_docs/version-0.15.0/faq.md
@@ -6,10 +6,10 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](faq_general)
-- [Design & Concepts](faq_design_and_concepts)
-- [Writing Tables](faq_writing_tables)
-- [Reading Tables](faq_reading_tables)
-- [Table Services](faq_table_services)
-- [Storage](faq_storage)
-- [Integrations](faq_integrations)
+- [General](faq_general.md)
+- [Design & Concepts](faq_design_and_concepts.md)
+- [Writing Tables](faq_writing_tables.md)
+- [Reading Tables](faq_reading_tables.md)
+- [Table Services](faq_table_services.md)
+- [Storage](faq_storage.md)
+- [Integrations](faq_integrations.md)
diff --git a/website/versioned_docs/version-0.15.0/faq_design_and_concepts.md b/website/versioned_docs/version-0.15.0/faq_design_and_concepts.md
index 720f4ae8e2df..8aee10defc1a 100644
--- a/website/versioned_docs/version-0.15.0/faq_design_and_concepts.md
+++ b/website/versioned_docs/version-0.15.0/faq_design_and_concepts.md
@@ -6,14 +6,14 @@ keywords: [hudi, writing, reading]
### How does Hudi ensure atomicity?
-Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
+Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline.md) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
### Does Hudi extend the Hive table layout?
Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
-* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
-* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users exper [...]
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
+* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering.md) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users ex [...]
* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudiβs design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
### What concurrency control approaches does Hudi adopt?
@@ -32,7 +32,7 @@ In these scenarios, it might be tempting to think of data inconsistencies/data l
(examples [1](https://github.com/apache/hudi/blob/aea5bb6f0ab824247f5e3498762ad94f643a2cb6/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L76),
[2](https://github.com/apache/hudi/blame/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L173)) in incremental queries to ensure that no data
is served beyond the point there is an inflight instant in its timeline, so no data loss or drop happens. This detection is made possible because Hudi writes first request a transaction on the timeline, before planning/executing
-the write, as explained in the [timeline](timeline#states) section.
+the write, as explained in the [timeline](timeline.md#states) section.
In this case, on seeing C1βs inflight commit (publish to timeline is atomic), C2 data (which is > C1 in the timeline) is not served until C1 inflight transitions to a terminal state such as completed or marked as failed.
This [test](https://github.com/apache/hudi/blob/master/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java#L137) demonstrates how Hudi incremental source stops proceeding until C1 completes.
@@ -48,7 +48,7 @@ To expand more on the long term approach, Hudi has had a proposal to streamline/
This has been delayed for a few reasons
- Large hosted query engines and users not upgrading fast enough.
-- The issues brought up - \[[1](faq_design_and_concepts#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
+- The issues brought up - \[[1](faq_design_and_concepts.md#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts.md#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
relevant to this are not practically very important to users beyond good pedantic discussions,
- Wanting to do it alongside [non-blocking concurrency control](https://github.com/apache/hudi/pull/7907) in Hudi version 1.x.
diff --git a/website/versioned_docs/version-0.15.0/faq_general.md b/website/versioned_docs/version-0.15.0/faq_general.md
index 3ded00b2a45f..5c44f8d6ebe7 100644
--- a/website/versioned_docs/version-0.15.0/faq_general.md
+++ b/website/versioned_docs/version-0.15.0/faq_general.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### When is Hudi useful for me or my organization?
-If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
+If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion.md). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
@@ -37,7 +37,7 @@ Choose Copy-on-write storage if :
* You are looking for a simple alternative, that replaces your existing parquet tables without any need for real-time data.
* Your current job is rewriting entire table/partition to deal with updates, while only a few files actually change in each partition.
-* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
+* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations.md#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
* Your workload is fairly well-understood and does not have sudden bursts of large amount of update or inserts to older partitions. COW absorbs all the merging cost on the writer side and thus these sudden changes can clog up your ingestion and interfere with meeting normal mode ingest latency targets.
Choose merge-on-read storage if :
@@ -51,7 +51,7 @@ Immaterial of what you choose, Hudi provides
* Incremental pulls
* Ability to de-duplicate data
-Find more [here](concepts/).
+Find more [here](concepts.md).
### Is Hudi an analytical database?
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data.md) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
@@ -79,7 +79,7 @@ generate the key internally (much like RDBMS row\_ids)
### How does Hudi actually store data inside a table?
-At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts/) for more details.
+At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts.md) for more details.
### How Hudi handles partition evolution requirements ?
diff --git a/website/versioned_docs/version-0.15.0/faq_reading_tables.md b/website/versioned_docs/version-0.15.0/faq_reading_tables.md
index 3f413b7ef1e0..e9db97dbbdd2 100644
--- a/website/versioned_docs/version-0.15.0/faq_reading_tables.md
+++ b/website/versioned_docs/version-0.15.0/faq_reading_tables.md
@@ -24,7 +24,7 @@ set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
### Does Hudi guarantee consistent reads? How to think about read optimized queries?
-Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
+Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types.md#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
Hudiβs read-optimized query is targeted for the MOR table only, with guidance around how compaction should be run to achieve predictable results. In the MOR table, the compaction, which runs every few commits (or βdeltacommitβ to be exact for the MOR table) by default, merges the base (parquet) file and corresponding change log files to a new base file within each file group, so that the snapshot query serving the latest data immediately after compaction reads the base files only.Β Simil [...]
diff --git a/website/versioned_docs/version-0.15.0/faq_storage.md b/website/versioned_docs/version-0.15.0/faq_storage.md
index 27fe9ed70ffd..d17e42003421 100644
--- a/website/versioned_docs/version-0.15.0/faq_storage.md
+++ b/website/versioned_docs/version-0.15.0/faq_storage.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### Does Hudi support cloud storage/object stores?
-Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
+Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud.md) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations.md#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
### What is the difference between copy-on-write (COW) vs merge-on-read (MOR) table types?
@@ -14,11 +14,11 @@ Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop
**Merge On Read** - This storage type enables clients to ingest data quickly onto row based data format such as avro. Any new data that is written to the Hudi table using MOR table type, will write new log/delta files that internally store the data as avro encoded bytes. A compaction process (configured as inline or asynchronous) will convert log file format to columnar file format (parquet). Two different InputFormats expose 2 different views of this data, Read Optimized view exposes co [...]
-More details can be found [here](concepts/) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
+More details can be found [here](concepts.md) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
### How do I migrate my data to Hudi?
-Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide/), including ways to doing partial migrations.
+Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables.md#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide.md), including ways to doing partial migrations.
### How to convert an existing COW table to MOR?
@@ -34,7 +34,7 @@ But manually changing it will result in checksum errors. So, we have to go via h
### How can I find the average record size in a commit?
-The `commit showpartitons` command in [HUDI CLI](cli) will show both "bytes written" and
+The `commit showpartitons` command in [HUDI CLI](cli.md) will show both "bytes written" and
"records inserted." Divide the bytes written by records inserted to find the average size. Note that this answer assumes
@@ -91,11 +91,11 @@ spark.read.parquet("your_data_set/path/to/month").limit(n) // Limit n records
.save(basePath);
```
-For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli).
+For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli.md).
### Why does maintain record level commit metadata? Isn't tracking table version at file level good enough?Β
-By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hudi [...]
+By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering.md#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hu [...]
### Why partition fields are also stored in parquet files in addition to the partition path ?
diff --git a/website/versioned_docs/version-0.15.0/faq_table_services.md b/website/versioned_docs/version-0.15.0/faq_table_services.md
index 7af1ec75398b..dc4dbcf9716f 100644
--- a/website/versioned_docs/version-0.15.0/faq_table_services.md
+++ b/website/versioned_docs/version-0.15.0/faq_table_services.md
@@ -6,11 +6,11 @@ keywords: [hudi, writing, reading]
### What does the Hudi cleaner do?
-The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be read [...]
+The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations.md#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be r [...]
### How do I run compaction for a MOR table?
-Simplest way to run compaction on MOR table is to run the [compaction inline](configurations#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted most o [...]
+Simplest way to run compaction on MOR table is to run the [compaction inline](configurations.md#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted mos [...]
That said, for obvious reasons of not blocking ingesting for compaction, you may want to run it asynchronously as well. This can be done either via a separate [compaction job](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java) that is scheduled by your workflow scheduler/notebook independently. If you are using Hudi Streamer, then you can run in [continuous mode](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca [...]
@@ -40,7 +40,7 @@ Depending on how you write to Hudi these are the possible options currently.
### How to disable all table services in case of multiple writers?
-[hoodie.table.services.enabled](configurations#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
+[hoodie.table.services.enabled](configurations.md#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
### Why does Hudi retain at-least one previous commit even after setting hoodie.cleaner.commits.retained': 1 ?
@@ -50,6 +50,6 @@ Hudi runs cleaner to remove old file versions as part of writing data either in
Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
-be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback)
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback.md)
for details
diff --git a/website/versioned_docs/version-0.15.0/faq_writing_tables.md b/website/versioned_docs/version-0.15.0/faq_writing_tables.md
index c08dfb92a5fa..51e92a8148c6 100644
--- a/website/versioned_docs/version-0.15.0/faq_writing_tables.md
+++ b/website/versioned_docs/version-0.15.0/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API and us [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations.md) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API a [...]
### How is a Hudi writer job deployed?
@@ -68,21 +68,21 @@ As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github
### How do I delete records in the dataset using Hudi?
-GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data#deletes).
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data.md#deletes).
### Should I need to worry about deleting all copies of the records in case of duplicates?
-No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https://githu [...]
+No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables.md#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables.md#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https:/ [...]
### How does Hudi handle duplicate record keys in an input?
-When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
+When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
-For an insert or bulk\_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or i [...]
+For an insert or bulk\_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) o [...]
### How can I pass hudi configurations to my spark writer job?
-Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations/). Invoking _\--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
+Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations.md). Invoking _\--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
* For Spark DataSource, you can use the "options" API of DataFrameWriter to pass in these configs.
@@ -108,28 +108,28 @@ hoodie.datasource.write.hive_style_partitioning: true
### Can I register my Hudi table with Apache Hive metastore?
-Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations#hoodiedatasourcehive_syncenable).
+Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore.md#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations.md#hoodiedatasourcehive_syncenable).
### What's Hudi's schema evolution story?
Hudi uses Avro as the internal canonical representation for records, primarily due to its nice [schema compatibility & evolution](https://docs.confluent.io/platform/current/schema-registry/avro.html) properties. This is a key aspect of having reliability in your ingestion or ETL pipelines. As long as the schema passed to Hudi (either explicitly in Hudi Streamer schema provider configs or implicitly by Spark Datasource's Dataset schemas) is backwards compatible (e.g no field deletes, only [...]
-Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution) for more details on Schema-on-read for Spark..
+Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution.md) for more details on Schema-on-read for Spark..
### What performance/ingest latency can I expect for Hudi writing?
-The speed at which you can write into Hudi depends on the [write operation](write_operations) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when try [...]
+The speed at which you can write into Hudi depends on the [write operation](write_operations.md) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when [...]
| Storage Type | Type of workload | Performance | Tips |
| ---| ---| ---| --- |
-| copy on write | bulk\_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism) to get right number of files. use insert if you want this auto tuned . Configure [hoodie.bulkinsert.sort.mode](configurations#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value NONE offers the fastest performance and matches `spark.write.parquet()` [...]
-| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
-| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or \<50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For e.g, if a partition has 1000 f [...]
+| copy on write | bulk\_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism) to get right number of files. use insert if you want this auto tuned . Configure [hoodie.bulkinsert.sort.mode](configurations.md#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value NONE offers the fastest performance and matches `spark.write.parqu [...]
+| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations.md#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
+| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or \<50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For e.g, if a partition has 1000 f [...]
| merge on read | bulk insert | Currently new data only goes to parquet files and thus performance here should be similar to copy\_on\_write bulk insert. This has the nice side-effect of getting data into parquet directly for query performance. [This GitHub issue](https://github.com/apache/hudi/issues/14468) will add support for logging inserts directly and this up drastically. | |
| merge on read | insert | Similar to above | |
| merge on read | upsert/ de-duplicate & insert | Indexing performance would remain the same as copy-on-write, while ingest latency for updates (costliest I/O operation in copy\_on\_write) are sent to log files and thus with asynchronous compaction provides very very good ingest performance with low write amplification. | |
-Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide) for more tips on JVM and other configurations.
+Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide.md) for more tips on JVM and other configurations.
### What performance can I expect for Hudi reading/queries?
@@ -147,11 +147,11 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk\_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations#hoodieparquetmaxfilesize) and the [soft limit](configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on previou [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations.md#hoodieparquetmaxfilesize) and the [soft limit](configurations.md#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on p [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
-* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations#hoodielogfilemaxsize) and a [factor](configurations#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
+* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations.md#hoodielogfilemaxsize) and a [factor](configurations.md#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
* Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations as above for the COPY\_ON\_WRITE case applies.
NOTE : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the compaction has been p [...]
@@ -173,7 +173,7 @@ hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPa
With each commit, Hudi creates a new table version in the metastore. This can be reduced by setting the option
-[hoodie.datasource.meta\_sync.condition.sync](configurations#hoodiedatasourcemeta_syncconditionsync) to true.
+[hoodie.datasource.meta\_sync.condition.sync](configurations.md#hoodiedatasourcemeta_syncconditionsync) to true.
This will ensure that hive sync is triggered on schema or partitions changes.
@@ -183,11 +183,11 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
+Hudi employs [optimistic concurrency control](concurrency_control.md) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/ [...]
### Can single-writer inserts have duplicates?
-By default, Hudi turns off key based de-duplication for INSERT/BULK\_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in delt [...]
+By default, Hudi turns off key based de-duplication for INSERT/BULK\_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in d [...]
### Can concurrent inserts cause duplicates?
diff --git a/website/versioned_docs/version-0.15.0/file_layouts.md b/website/versioned_docs/version-0.15.0/file_layouts.md
index 6fca19dec491..d510f9e7d38b 100644
--- a/website/versioned_docs/version-0.15.0/file_layouts.md
+++ b/website/versioned_docs/version-0.15.0/file_layouts.md
@@ -10,8 +10,8 @@ The following describes the general file layout structure for Apache Hudi. Pleas
* Each file group contains several file slices
* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
-Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
-file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner.md) action gets rid of unused/older file slices to reclaim space on the file system.
diff --git a/website/versioned_docs/version-0.15.0/file_sizing.md b/website/versioned_docs/version-0.15.0/file_sizing.md
index a451b09b6c58..61356932afbb 100644
--- a/website/versioned_docs/version-0.15.0/file_sizing.md
+++ b/website/versioned_docs/version-0.15.0/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner.md)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md
index 36546d0052b6..ada109209eae 100644
--- a/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md
@@ -59,7 +59,7 @@ quick start tool for SQL users.
### Start Flink SQL client
Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
-You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore#install)), or download it from the
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
Now start the SQL CLI:
@@ -281,7 +281,7 @@ Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-e
>
This statement queries snapshot view of the dataset.
-Refers to [Table types and queries](concepts#table-types--queries) for more info on all table types and query types supported.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
## Update Data
@@ -324,7 +324,7 @@ Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/ma
</Tabs
>
-[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts)
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
denoted by the timestamp.
@@ -448,19 +448,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink#cdc-ingestion), [Bulk Insert](ingestion_flink#bulk-insert), [Index Bootstrap](ingestion_flink#index-bootstrap), [Changelog Mode](ingestion_flink#changelog-mode) and [Append Mode](ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml#non-blocking-concurrency-control-experimental).
-- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](sql_queries#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
-- **Optimization**: Offline compaction is supported [Offline Compaction](compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk Insert](ingestion_flink.md#bulk-insert), [Index Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog Mode](ingestion_flink.md#changelog-mode) and [Append Mode](ingestion_flink.md#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml.md#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](sql_queries.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
- - [Hudi File Layout](file_layouts) - How the files are laid out on storage
- - [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+ - [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -469,5 +469,5 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it. Also, if you are looking for ways to migrate your existing data
-to Hudi, refer to [migration guide](migration_guide).
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-0.15.0/hoodie_cleaner.md b/website/versioned_docs/version-0.15.0/hoodie_cleaner.md
index d91d5c6c5800..dcaac429bf8d 100644
--- a/website/versioned_docs/version-0.15.0/hoodie_cleaner.md
+++ b/website/versioned_docs/version-0.15.0/hoodie_cleaner.md
@@ -126,7 +126,7 @@ spark-submit --master local --class org.apache.hudi.utilities.HoodieCleaner `ls
Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
#### CLI
-You can also use [Hudi CLI](cli) to run Hoodie Cleaner.
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
CLI provides the below commands for cleaner service:
- `cleans show`
diff --git a/website/versioned_docs/version-0.15.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-0.15.0/hoodie_streaming_ingestion.md
index f8bfdfbfa175..1158c695df6c 100644
--- a/website/versioned_docs/version-0.15.0/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-0.15.0/hoodie_streaming_ingestion.md
@@ -264,7 +264,7 @@ and then ingest it as follows.
--op BULK_INSERT
```
-In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide).
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
### Using `hudi-utilities` bundle jars
@@ -302,7 +302,7 @@ A Hudi Streamer job can then be triggered as follows:
--op BULK_INSERT
```
-Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control) section
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
### Checkpointing
diff --git a/website/versioned_docs/version-0.15.0/hudi_stack.md b/website/versioned_docs/version-0.15.0/hudi_stack.md
index af8bfb8396ec..dfe2a46a2081 100644
--- a/website/versioned_docs/version-0.15.0/hudi_stack.md
+++ b/website/versioned_docs/version-0.15.0/hudi_stack.md
@@ -32,19 +32,19 @@ The transactional database layer of Hudi comprises the core components that are
Drawing an analogy to file formats, a table format simply comprises the file layout of the table, the schema, and metadata tracking changes. Hudi organizes files within a table or partition into File Groups. Updates are captured in Log Files tied to these File Groups, ensuring efficient merges. There are three major components related to Hudiβs table format.
-- **Timeline** : Hudi's [timeline](timeline), stored in the /.hoodie folder, is a crucial event log recording all table actions in an ordered manner, with events kept for a specified period. Hudi uniquely designs each File Group as a self-contained log, enabling record state reconstruction through delta logs, even after archival of related actions. This approach effectively limits metadata size based on table activity frequency, essential for managing tables with frequent updates.
+- **Timeline** : Hudi's [timeline](timeline.md), stored in the /.hoodie folder, is a crucial event log recording all table actions in an ordered manner, with events kept for a specified period. Hudi uniquely designs each File Group as a self-contained log, enabling record state reconstruction through delta logs, even after archival of related actions. This approach effectively limits metadata size based on table activity frequency, essential for managing tables with frequent updates.
- **File Group and File Slice** : Within each partition the data is physically stored as base and Log Files and organized into logical concepts as [File groups](https://hudi.apache.org/tech-specs-1point0/#storage-layout) and File Slices. File groups contain multiple versions of File Slices and are split into multiple File Slices. A File Slice comprises the Base and Log File. Each File Slice within the file-group is uniquely identified by the commit's timestamp that created it.
-- **Metadata Table** : Implemented as a merge-on-read table, Hudi's [metadata table](metadata) efficiently handles quick updates with low write amplification. It leverages the HFile format for quick, indexed key lookups, storing vital information like file paths, column statistics, bloom filters, and record indexes. This approach streamlines operations by reducing the necessity for expensive cloud file listings. The metadata table in Hudi acts as an additional [indexing system](metadata# [...]
+- **Metadata Table** : Implemented as a merge-on-read table, Hudi's [metadata table](metadata.md) efficiently handles quick updates with low write amplification. It leverages the HFile format for quick, indexed key lookups, storing vital information like file paths, column statistics, bloom filters, and record indexes. This approach streamlines operations by reducing the necessity for expensive cloud file listings. The metadata table in Hudi acts as an additional [indexing system](metada [...]
-Hudiβs approach of recording updates into Log Files is more efficient and involves low merge overhead than systems like Hive ACID, where merging all delta records against all Base Files is required. Read more about the various table types in Hudi [here](table_types).
+Hudiβs approach of recording updates into Log Files is more efficient and involves low merge overhead than systems like Hive ACID, where merging all delta records against all Base Files is required. Read more about the various table types in Hudi [here](table_types.md).
### Indexes
<p align = "center">Figure: Indexes in Hudi</p>
-[Indexes](indexing) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. Hudiβs [metadata table](metadata/#metadata-table-indices) brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata table, like file listings, column statistics, bloom filters, record-level indexes, and [functional indexes](https://github.com/apache/hudi/blob/master/rfc [...]
+[Indexes](indexing.md) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. Hudiβs [metadata table](metadata/#metadata-table-indices) brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata table, like file listings, column statistics, bloom filters, record-level indexes, and [functional indexes](https://github.com/apache/hudi/blob/master/ [...]
### Table Services
@@ -53,19 +53,19 @@ Hudiβs approach of recording updates into Log Files is more efficient and invo
Apache Hudi offers various table services to help keep the table storage layout and metadata management performant. Hudi was designed with built-in table services that enables running them in inline, semi-asynchronous or full-asynchronous modes. Furthermore, Spark and Flink streaming writers can run in continuous mode, and invoke table services asynchronously sharing the underlying executors intelligently with writers. Letβs take a look at these services.
#### Clustering
-The [clustering](clustering) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into larger ones for optimal file size management. It's fully integrated with other timeline actions like cleaning and compaction, enabling smart optimizations such as avoiding compaction for File Groups undergoing clustering, thereby saving on I/O.
+The [clustering](clustering.md) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into larger ones for optimal file size management. It's fully integrated with other timeline actions like cleaning and compaction, enabling smart optimizations such as avoiding compaction for File Groups undergoing clustering, thereby saving on I/O.
#### Compaction
-Hudi's [compaction](compaction) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows concurrent writes to the same File Froup, enabled by Hudi's file grouping and flexible log merging. This facilitates non-blocking execution of deletes even during concurrent record updates.
+Hudi's [compaction](compaction.md) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows concurrent writes to the same File Froup, enabled by Hudi's file grouping and flexible log merging. This facilitates non-blocking execution of deletes even during concurrent record updates.
#### Cleaning
[Cleaner](http://hudi.apache.org/blog/2021/06/10/employing-right-configurations-for-hudi-cleaner) service works off the timeline incrementally, removing File Slices that are past the configured retention period for incremental queries, while also allowing sufficient time for long running batch jobs (e.g Hive ETLs) to finish running. This allows users to reclaim storage space, thereby saving on costs.
#### Indexing
-Hudi's scalable metadata table contains auxiliary data about the table. This subsystem encompasses various indices, including files, column_stats, and bloom_filters, facilitating efficient record location and data skipping. Balancing write throughput with index updates presents a fundamental challenge, as traditional indexing methods, like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchro [...]
+Hudi's scalable metadata table contains auxiliary data about the table. This subsystem encompasses various indices, including files, column_stats, and bloom_filters, facilitating efficient record location and data skipping. Balancing write throughput with index updates presents a fundamental challenge, as traditional indexing methods, like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchro [...]
### Concurrency Control
-[Concurrency control](concurrency_control) defines how different writers/readers coordinate access to the table. Hudiβs way of publishing monotonically increasing timestamped commits to the timeline lays out the foundation for atomicity guarantees, clearly differentiating between writers (responsible for upserts/deletes), table services (focusing on storage optimization and bookkeeping), and readers (for query execution). Hudi provides snapshot isolation, offering a consistent view of th [...]
+[Concurrency control](concurrency_control.md) defines how different writers/readers coordinate access to the table. Hudiβs way of publishing monotonically increasing timestamped commits to the timeline lays out the foundation for atomicity guarantees, clearly differentiating between writers (responsible for upserts/deletes), table services (focusing on storage optimization and bookkeeping), and readers (for query execution). Hudi provides snapshot isolation, offering a consistent view of [...]
### Lake Cache*
@@ -93,7 +93,7 @@ Hudi provides snapshot isolation for writers and readers, enabling consistent ta
<p align = "center">Figure: Various platform services in Hudi</p>
-Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers. Services, like [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (including [...]
+Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers. Services, like [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (includi [...]
### Query Engines
Apache Hudi is compatible with a wide array of query engines, catering to various analytical needs. For distributed ETL batch processing, Apache Spark is frequently utilized, leveraging its efficient handling of large-scale data. In the realm of streaming use cases, compute engines such as Apache Flink and Apache Spark's Structured Streaming provide robust support when paired with Hudi. Moreover, Hudi supports modern data lake query engines such as Trino and Presto, as well as modern ana [...]
diff --git a/website/versioned_docs/version-0.15.0/indexing.md b/website/versioned_docs/version-0.15.0/indexing.md
index 4a233123530d..fb45650a0f19 100644
--- a/website/versioned_docs/version-0.15.0/indexing.md
+++ b/website/versioned_docs/version-0.15.0/indexing.md
@@ -11,9 +11,9 @@ Hudi provides efficient upserts, by mapping a given hoodie key (record key + par
This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
mapped file group contains all versions of a group of records.
-For [Copy-On-Write tables](table_types#copy-on-write-table), this enables fast upsert/delete operations, by
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), this enables fast upsert/delete operations, by
avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](table_types#merge-on-read-table), this design allows Hudi to bound the amount of
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), this design allows Hudi to bound the amount of
records any given base file needs to be merged against.
Specifically, a given base file needs to merged only against updates for records that are part of that base file. In contrast,
designs without an indexing component (e.g: [Apache Hive ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)),
diff --git a/website/versioned_docs/version-0.15.0/ingestion_flink.md b/website/versioned_docs/version-0.15.0/ingestion_flink.md
index 8176ff1e5ac6..8ec300910490 100644
--- a/website/versioned_docs/version-0.15.0/ingestion_flink.md
+++ b/website/versioned_docs/version-0.15.0/ingestion_flink.md
@@ -37,7 +37,7 @@ frequent `file handle` switching.
:::note
The parallelism of `bulk_insert` is specified by `write.tasks`. The parallelism will affect the number of small files.
In theory, the parallelism of `bulk_insert` is the number of `bucket`s (In particular, when each bucket writes to maximum file size, it
-will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks).
+will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks).
:::
#### Options
@@ -45,7 +45,7 @@ will rollover to the new file handle. Finally, `the number of files` >= [`write.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.operation` | `true` | `upsert` | Setting as `bulk_insert` to open this function |
-| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks) |
+| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks) |
| `write.bulk_insert.shuffle_input` | `false` | `true` | Whether to shuffle data according to the input field before writing. Enabling this option will reduce the number of small files, but there may be a risk of data skew |
| `write.bulk_insert.sort_input` | `false` | `true` | Whether to sort data according to the input field before writing. Enabling this option will reduce the number of small files when a write task writes multiple partitions |
| `write.sort.memory` | `false` | `128` | Available managed memory of sort operator. default `128` MB |
diff --git a/website/versioned_docs/version-0.15.0/metadata.md b/website/versioned_docs/version-0.15.0/metadata.md
index 69b1929377de..97b3fdb0208c 100644
--- a/website/versioned_docs/version-0.15.0/metadata.md
+++ b/website/versioned_docs/version-0.15.0/metadata.md
@@ -106,7 +106,7 @@ Following are the Spark based basic configs that are needed to enable metadata a
The metadata table with synchronous updates and metadata-table-based file listing are enabled by default.
There are prerequisite configurations and steps in [Deployment considerations](#deployment-considerations-for-metadata-table) to
safely use this feature. The metadata table and related file listing functionality can still be turned off by setting
-[`hoodie.metadata.enable`](configurations#hoodiemetadataenable) to `false`. The
+[`hoodie.metadata.enable`](configurations.md#hoodiemetadataenable) to `false`. The
[multi-modal index](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi) are
disabled by default and can be enabled in write side explicitly using the above configs.
@@ -151,17 +151,17 @@ corresponding configs across Spark and Flink readers.
## Deployment considerations for metadata Table
To ensure that metadata table stays up to date, all write operations on the same Hudi table need additional configurations
besides the above in different deployment models. Before enabling metadata table, all writers on the same table must
-be stopped. Please refer to the different [deployment models](concurrency_control#deployment-models-with-supported-concurrency-controls)
+be stopped. Please refer to the different [deployment models](concurrency_control.md#deployment-models-with-supported-concurrency-controls)
for more details on each model. This section only highlights how to safely enable metadata table in different deployment models.
### Deployment Model A: Single writer with inline table services
-In [Model A](concurrency_control#model-a-single-writer-with-inline-table-services), after setting [`hoodie.metadata.enable`](configurations#hoodiemetadataenable) to `true`, restarting
+In [Model A](concurrency_control.md#model-a-single-writer-with-inline-table-services), after setting [`hoodie.metadata.enable`](configurations.md#hoodiemetadataenable) to `true`, restarting
the single writer is sufficient to safely enable metadata table.
### Deployment Model B: Single writer with async table services
-If your current deployment model is [Model B](concurrency_control#model-b-single-writer-with-async-table-services), enabling metadata
+If your current deployment model is [Model B](concurrency_control.md#model-b-single-writer-with-async-table-services), enabling metadata
table requires adding optimistic concurrency control along with suggested lock provider like below.
```properties
hoodie.write.concurrency.mode=optimistic_concurrency_control
@@ -179,7 +179,7 @@ process which cannot rely on the in-process lock provider.
### Deployment Model C: Multi-writer
-If your current deployment model is [multi-writer](concurrency_control#model-c-multi-writer) along with a lock
+If your current deployment model is [multi-writer](concurrency_control.md#model-c-multi-writer) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only partial writers leads to loss of data from the inconsistent writer. So, ensure you enable
@@ -190,7 +190,7 @@ hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.write.lock.provider=<distributed-lock-provider-classname>
```
-Note that there are different external [lock providers available](concurrency_control#external-locking-and-lock-providers)
+Note that there are different external [lock providers available](concurrency_control.md#external-locking-and-lock-providers)
to choose from.
## Related Resources
diff --git a/website/versioned_docs/version-0.15.0/metadata_indexing.md b/website/versioned_docs/version-0.15.0/metadata_indexing.md
index 54a446a0b7ed..b96b53025456 100644
--- a/website/versioned_docs/version-0.15.0/metadata_indexing.md
+++ b/website/versioned_docs/version-0.15.0/metadata_indexing.md
@@ -5,7 +5,7 @@ toc: true
last_modified_at:
---
-Hudi maintains a scalable [metadata](metadata) that has some auxiliary data about the table.
+Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data about the table.
The [pluggable indexing subsystem](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi)
of Hudi depends on the metadata table. Different types of index, from `files` index for locating records efficiently
to `column_stats` index for data skipping, are part of the metadata table. A fundamental tradeoff in any data system
@@ -25,7 +25,7 @@ feature, please check out [this blog](https://www.onehouse.ai/blog/asynchronous-
## Setup Async Indexing
-First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) which will continuously write data
+First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) which will continuously write data
from raw parquet to Hudi table. We used the widely available [NY Taxi dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/), whose setup details are as below:
<details>
<summary>Ingestion write config</summary>
@@ -78,7 +78,7 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](metadata#metadata-table-indices) along with [configs](metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
+metadata table can be explored [here](metadata.md#metadata-table-indices) along with [configs](metadata.md#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
diff --git a/website/versioned_docs/version-0.15.0/metrics.md b/website/versioned_docs/version-0.15.0/metrics.md
index c6f2833f5ada..c389689716be 100644
--- a/website/versioned_docs/version-0.15.0/metrics.md
+++ b/website/versioned_docs/version-0.15.0/metrics.md
@@ -6,7 +6,7 @@ toc: true
last_modified_at: 2020-06-20T15:59:57-04:00
---
-In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations#METRICS).
+In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations.md#METRICS).
## MetricsReporter
@@ -17,7 +17,7 @@ MetricsReporter provides APIs for reporting `HoodieMetrics` to user-specified ba
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
#### Configurations
-The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Jmx).
+The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Jmx).
```properties
hoodie.metrics.on=true
@@ -37,7 +37,7 @@ As configured above, JmxMetricsReporter will started JMX server on port 4001. We
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send `HoodieMetrics` to it.
#### Configurations
-The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations#Metrics-Configurations-for-Graphite).
+The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Graphite).
```properties
hoodie.metrics.on=true
@@ -58,7 +58,7 @@ DatadogMetricsReporter is an implementation of Datadog reporter.
A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
#### Configurations
-The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Datadog-reporter).
+The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Datadog-reporter).
```properties
hoodie.metrics.on=true
@@ -138,7 +138,7 @@ tuned are in the `HoodieMetricsCloudWatchConfig` class.
Allows users to define a custom metrics reporter.
#### Configurations
-The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations).
+The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations).
```properties
hoodie.metrics.on=true
diff --git a/website/versioned_docs/version-0.15.0/migration_guide.md b/website/versioned_docs/version-0.15.0/migration_guide.md
index 2baf7ee0dc0f..b63bc3fbd596 100644
--- a/website/versioned_docs/version-0.15.0/migration_guide.md
+++ b/website/versioned_docs/version-0.15.0/migration_guide.md
@@ -40,7 +40,7 @@ There are a few options when choosing this approach.
#### Using Hudi Streamer
-Use the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool. HoodieStreamer supports bootstrap with
+Use the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool. HoodieStreamer supports bootstrap with
--run-bootstrap command line option. There are two types of bootstrap, METADATA_ONLY and FULL_RECORD. METADATA_ONLY will
generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will
perform a full copy/rewrite of the data as a Hudi table. Additionally, once can choose selective partitions using regex
@@ -88,7 +88,7 @@ Refer to [Bootstrap procedure](https://hudi.apache.org/docs/next/procedures#boot
#### Using Hudi CLI
Write your own custom logic of how to load an existing table into a Hudi managed one. Please read about the RDD API
-[here](quick-start-guide). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
+[here](quick-start-guide.md). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hudi-cli && ./hudi-cli.sh`.
```java
diff --git a/website/versioned_docs/version-0.15.0/overview.mdx b/website/versioned_docs/version-0.15.0/overview.mdx
index 96a64f2fc695..defdd2c75dbd 100644
--- a/website/versioned_docs/version-0.15.0/overview.mdx
+++ b/website/versioned_docs/version-0.15.0/overview.mdx
@@ -13,18 +13,18 @@ how to learn more to get started.
## What is Apache Hudi
Apache Hudi (pronounced βhoodieβ) is the next generation [streaming data lake platform](/blog/2021/07/21/streaming-data-lake-platform).
- Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](sql_ddl),
-[transactions](timeline), [efficient upserts/deletes](write_operations), [advanced indexes](indexing),
-[ingestion services](hoodie_streaming_ingestion), data [clustering](clustering)/[compaction](compaction) optimizations,
-and [concurrency](concurrency_control) all while keeping your data in open source file formats.
+ Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides [tables](sql_ddl.md),
+[transactions](timeline.md), [efficient upserts/deletes](write_operations.md), [advanced indexes](indexing.md),
+[ingestion services](hoodie_streaming_ingestion.md), data [clustering](clustering.md)/[compaction](compaction.md) optimizations,
+and [concurrency](concurrency_control.md) all while keeping your data in open source file formats.
Not only is Apache Hudi great for streaming workloads, but it also allows you to create efficient incremental batch pipelines.
-Read the docs for more [use case descriptions](use_cases) and check out [who's using Hudi](/powered-by), to see how some of the
+Read the docs for more [use case descriptions](use_cases.md) and check out [who's using Hudi](/powered-by), to see how some of the
largest data lakes in the world including [Uber](https://www.uber.com/en-IN/blog/uber-big-data-platform/), [Amazon](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/),
[ByteDance](http://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance),
[Robinhood](https://s.apache.org/hudi-robinhood-talk) and more are transforming their production data lakes with Hudi.
-Apache Hudi can easily be used on any [cloud storage platform](cloud).
+Apache Hudi can easily be used on any [cloud storage platform](cloud.md).
Hudiβs advanced performance optimizations, make analytical workloads faster with any of
the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.
@@ -34,10 +34,10 @@ expands the use of Apache Hudi for a diverse range of use cases in the non-JVM e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
-- [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
-- [Hudi File Layout](file_layouts) - How the files are laid out on storage
-- [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
-- [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+- [Hudi File Layout](file_layouts.md) - How the files are laid out on storage
+- [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+- [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -47,13 +47,13 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:
-- [Spark Quick Start Guide](quick-start-guide) β if you primarily use Apache Spark
-- [Flink Quick Start Guide](flink-quick-start-guide) β if you primarily use Apache Flink
-- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide) - if you primarily use Python or Rust
+- [Spark Quick Start Guide](quick-start-guide.md) β if you primarily use Apache Spark
+- [Flink Quick Start Guide](flink-quick-start-guide.md) β if you primarily use Apache Flink
+- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide.md) - if you primarily use Python or Rust
If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the Docker Demo:
-- [Docker Demo](docker_demo)
+- [Docker Demo](docker_demo.md)
## Connect With The Community
Apache Hudi is community focused and community led and welcomes new-comers with open arms. Leverage the following
diff --git a/website/versioned_docs/version-0.15.0/performance.md b/website/versioned_docs/version-0.15.0/performance.md
index cd9de72d5a08..9fec829e492f 100644
--- a/website/versioned_docs/version-0.15.0/performance.md
+++ b/website/versioned_docs/version-0.15.0/performance.md
@@ -16,12 +16,12 @@ Here are some ways to efficiently manage the storage of your Hudi tables.
- The [small file handling feature](configurations/#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](configurations#hoodieparquetcompressionratio),
+- Cleaner can be [configured](configurations.md#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](configurations.md#hoodieparquetmaxfilesize), [log files](configurations.md#hoodielogfilemaxsize) & expected [compression ratio](configurations.md#hoodieparquetcompressionratio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
-- Intelligently tuning the [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
+- Intelligently tuning the [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be changed without re-clustering the table. Writes will simply expand given file groups with new updates/inserts as explained before.
-- For workloads with heavy updates, the [merge-on-read table](concepts#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
+- For workloads with heavy updates, the [merge-on-read table](concepts.md#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
## Performance Optimizations
@@ -119,11 +119,11 @@ For very large tables (1Tb+, 10s of 1000s of files), Data skipping could
To unlock the power of Data Skipping you will need to
-1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
+1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing.md)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
2. Enable Data Skipping in your queries, using `hoodie.metadata.index.column.stats.enable=true` (to enable Column Stats Index being populated on the write path, disabled by default)
:::note
-If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
+If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing.md) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
:::
To enable Data Skipping in your queries make sure to set following properties to `true` (on the read path):
diff --git a/website/versioned_docs/version-0.15.0/precommit_validator.md b/website/versioned_docs/version-0.15.0/precommit_validator.md
index ada26dd7a4c8..47a8add2696a 100644
--- a/website/versioned_docs/version-0.15.0/precommit_validator.md
+++ b/website/versioned_docs/version-0.15.0/precommit_validator.md
@@ -8,7 +8,7 @@ Data quality refers to the overall accuracy, completeness, consistency, and vali
Hudi offers **Pre-Commit Validators** that allow you to ensure that your data meets certain data quality expectations as you are writing with Hudi Streamer or Spark Datasource writers.
:::note
-Pre-commit validators are skipped when using the [BULK_INSERT](write_operations#bulk_insert) write operation type.
+Pre-commit validators are skipped when using the [BULK_INSERT](write_operations.md#bulk_insert) write operation type.
:::
Multiple class names can be separated by `,` delimiter.
@@ -105,7 +105,7 @@ void validateRecordsBeforeAndAfter(Dataset<Row> before,
```
## Additional Monitoring with Notifications
-Hudi offers a [commit notification service](platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
+Hudi offers a [commit notification service](platform_services_post_commit_callback.md) that can be configured to trigger notifications about write commits.
The commit notification service can be combined with pre-commit validators to send a notification when a commit fails a validation. This is possible by passing details about the validation as a custom value to the HTTP endpoint.
diff --git a/website/versioned_docs/version-0.15.0/procedures.md b/website/versioned_docs/version-0.15.0/procedures.md
index 938ec385d8a3..63abf83c1a27 100644
--- a/website/versioned_docs/version-0.15.0/procedures.md
+++ b/website/versioned_docs/version-0.15.0/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations.md#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations.md#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations.md#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations.md#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1533,11 +1533,11 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
-| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
-| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [clean_policy](configurations.md#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum qu [...]
+| [retain_commits](configurations.md#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations.md#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of [...]
+| [file_versions_retained](configurations.md#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations.md#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
@@ -1633,12 +1633,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations.md#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations.md#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations.md#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations.md#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations.md#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations.md#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1848,18 +1848,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations.md#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations.md#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations.md#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
+| [parallelism](configurations.md#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-0.15.0/python-rust-quick-start-guide.md b/website/versioned_docs/version-0.15.0/python-rust-quick-start-guide.md
index ab1eedfd36a4..2d91de30a890 100644
--- a/website/versioned_docs/version-0.15.0/python-rust-quick-start-guide.md
+++ b/website/versioned_docs/version-0.15.0/python-rust-quick-start-guide.md
@@ -22,7 +22,7 @@ cargo add hudi
> [!NOTE]
> These examples expect a Hudi table exists at `/tmp/trips_table`, created using
-> the [quick start guide](quick-start-guide).
+> the [quick start guide](quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-0.15.0/querying_data.md b/website/versioned_docs/version-0.15.0/querying_data.md
index c5ff60764839..7f23b02a0af1 100644
--- a/website/versioned_docs/version-0.15.0/querying_data.md
+++ b/website/versioned_docs/version-0.15.0/querying_data.md
@@ -7,10 +7,10 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl.md), [SQL DML](sql_dml.md), [SQL Queries](sql_queries.md) and [Procedures](procedures.md) for the latest documentation.
:::
-Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts#query-types).
+Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts.md#query-types).
Once the table is synced to the Hive metastore, it provides external Hive tables backed by Hudi's custom inputformats. Once the proper hudi
bundle has been installed, the table can be queried by popular query engines like Hive, Spark SQL, Flink, Trino and PrestoDB.
@@ -19,7 +19,7 @@ In sections, below we will discuss specific setup to access different query type
## Spark Datasource
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
-See the [Spark Quick Start](quick-start-guide) for more examples of Spark datasource reading queries.
+See the [Spark Quick Start](quick-start-guide.md) for more examples of Spark datasource reading queries.
**Setup**
@@ -56,8 +56,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Incremental Queries](quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](configurations#SPARK_DATASOURCE) section, to view all datasource options.
+For examples, refer to [Incremental Queries](quick-start-guide.md#incremental-query) in the Spark quickstart.
+Please refer to [configurations](configurations.md#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.15.0/quick-start-guide.md b/website/versioned_docs/version-0.15.0/quick-start-guide.md
index 0f8f362d117d..85f0d4988798 100644
--- a/website/versioned_docs/version-0.15.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.15.0/quick-start-guide.md
@@ -274,7 +274,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl.md) reference guide.
</TabItem>
@@ -318,7 +318,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -351,7 +351,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -360,7 +360,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml.md#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -472,7 +472,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml.md#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -502,7 +502,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts.md).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -590,7 +590,7 @@ MERGE statement either using `SET *` or using `SET column1 = expression1 [, colu
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data.md#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -960,7 +960,7 @@ Note that CDC queries are currently only supported on Copy-on-Write tables.
The examples thus far have showcased one of the two table types, that Hudi supports - Copy-on-Write (COW) tables. Hudi also supports
a more advanced write-optimized table type called Merge-on-Read (MOR) tables, that can balance read and write performance in a more
-flexible manner. See [table types](table_types) for more details.
+flexible manner. See [table types](table_types.md) for more details.
Any of these examples can be run on a Merge-on-Read table by simply changing the table type to MOR, while creating the table, as below.
@@ -1086,7 +1086,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations.md).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1159,29 +1159,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide.md).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
-For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl.md) and [Spark SQL DML](sql_dml.md) reference guides.
+For alter table commands, check out [this](sql_ddl.md#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures.md) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion.md#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes#spark-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes.md#spark-streaming) for more.
-Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data) and [streaming writes](writing_tables_streaming_writes).
+Check out more information on [modeling data in Hudi](faq_general.md#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data.md) and [streaming writes](writing_tables_streaming_writes.md).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it.
+steps [here](docker_demo.md) to get a taste for it.
diff --git a/website/versioned_docs/version-0.15.0/record_payload.md b/website/versioned_docs/version-0.15.0/record_payload.md
index 50f6f6ec57a5..49848a0d41f9 100644
--- a/website/versioned_docs/version-0.15.0/record_payload.md
+++ b/website/versioned_docs/version-0.15.0/record_payload.md
@@ -105,7 +105,7 @@ hoodie.datasource.write.payload.class=org.apache.hudi.common.model.EventTimeAvro
```
This is the default record payload for Flink based writing. Some use cases require merging records by event time and
thus event time plays the role of an ordering field. This payload is particularly useful in the case of late-arriving data.
-For such use cases, users need to set the [payload event time field](configurations#RECORD_PAYLOAD) configuration.
+For such use cases, users need to set the [payload event time field](configurations.md#RECORD_PAYLOAD) configuration.
#### OverwriteNonDefaultsWithLatestAvroPayload
```scala
@@ -171,7 +171,7 @@ provides support for seamlessly applying changes captured via Debezium for MySQL
provides support for applying changes captured via Amazon Database Migration Service onto S3.
Record payloads are tunable to suit many use cases. Please check out the configurations
-listed [here](configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
+listed [here](configurations.md#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.15.0/s3_hoodie.md b/website/versioned_docs/version-0.15.0/s3_hoodie.md
index 5faad6e62be9..fe3c9e513cfe 100644
--- a/website/versioned_docs/version-0.15.0/s3_hoodie.md
+++ b/website/versioned_docs/version-0.15.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](hoodie_cleaner) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](hoodie_cleaner.md) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.0/sql_ddl.md b/website/versioned_docs/version-0.15.0/sql_ddl.md
index c6cf5d90d9e0..831a9793759e 100644
--- a/website/versioned_docs/version-0.15.0/sql_ddl.md
+++ b/website/versioned_docs/version-0.15.0/sql_ddl.md
@@ -75,7 +75,7 @@ should be specified as `PARTITIONED BY (dt, hh)`.
### Create table with record keys and ordering fields
-As discussed [here](quick-start-guide#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
Typically, this is also accompanied by configuring a `preCombineField` option to deal with out-of-order data and potential
duplicate records with the same key in the incoming writes.
@@ -104,7 +104,7 @@ TBLPROPERTIES (
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion.md#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -378,7 +378,7 @@ Users can set table properties while creating a table. The important table prope
| Parameter Name | Default | Description |
|------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types) |
+| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types.md) |
| primaryKey | uuid | The primary key field names of the table separated by commas. Same as `hoodie.datasource.write.recordkey.field`. If this config is ignored, hudi will auto-generate primary keys. If explicitly set, primary key generation will honor user configuration. |
| preCombineField | | The pre-combine field of the table. It is used for resolving the final version of the record among multiple versions. Generally, `event time` or another similar column will be used for ordering purposes. Hudi will be able to handle out-of-order data using the preCombine field value. |
@@ -389,7 +389,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control.md#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -612,7 +612,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations.md#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-0.15.0/sql_dml.md b/website/versioned_docs/version-0.15.0/sql_dml.md
index 81171e442a5f..02df525335c5 100644
--- a/website/versioned_docs/version-0.15.0/sql_dml.md
+++ b/website/versioned_docs/version-0.15.0/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
### Insert Into
@@ -25,7 +25,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -45,7 +45,7 @@ INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000
```
:::note Mapping to write operations
-Hudi offers flexibility in choosing the underlying [write operation](write_operations) of a `INSERT INTO` statement using
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
and *"upsert"* (with deduplication/merging). If a precombine field is not set, *"insert"* is chosen as the default. For a table with preCombine field set,
*"upsert"* is chosen as the default operation.
@@ -317,7 +317,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control.md#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-0.15.0/sql_queries.md b/website/versioned_docs/version-0.15.0/sql_queries.md
index 856433121654..2943a98e43ed 100644
--- a/website/versioned_docs/version-0.15.0/sql_queries.md
+++ b/website/versioned_docs/version-0.15.0/sql_queries.md
@@ -8,11 +8,11 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-Hudi stores and organizes data on storage while providing different ways of [querying](concepts#query-types), across a wide range of query engines.
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
This page will show how to issue different queries and discuss any specific instructions for each query engine.
## Spark SQL
-The Spark [quickstart](quick-start-guide) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
@@ -91,11 +91,11 @@ For e.g the table has received 10 million modifications across 1 million records
see all changes in a given time window and not just the latest values.
:::
-Please refer to [configurations](basic_configurations) section for the important configuration options.
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
## Flink SQL
Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
-relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide).
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-0.15.0/structure.md b/website/versioned_docs/version-0.15.0/structure.md
index 31281b68f71c..9f78f89f16f5 100644
--- a/website/versioned_docs/version-0.15.0/structure.md
+++ b/website/versioned_docs/version-0.15.0/structure.md
@@ -17,4 +17,4 @@ Hudi (pronounced βHoodieβ) ingests & manages storage of large analytical tab
By carefully managing how data is laid out in storage & how itβs exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [PrestoDB](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) [...]
-Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide) for a demo.
+Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide.md) for a demo.
diff --git a/website/versioned_docs/version-0.15.0/syncing_aws_glue_data_catalog.md b/website/versioned_docs/version-0.15.0/syncing_aws_glue_data_catalog.md
index b6f6c82a6c54..35f43a8af472 100644
--- a/website/versioned_docs/version-0.15.0/syncing_aws_glue_data_catalog.md
+++ b/website/versioned_docs/version-0.15.0/syncing_aws_glue_data_catalog.md
@@ -10,7 +10,7 @@ and send them to AWS Glue.
## Configurations
Most of the configurations for `AwsGlueCatalogSyncTool` are shared with `HiveSyncTool`. The example showed in
-[Sync to Hive Metastore](syncing_metastore) can be used as is for sync with Glue Data Catalog, provided that the hive metastore
+[Sync to Hive Metastore](syncing_metastore.md) can be used as is for sync with Glue Data Catalog, provided that the hive metastore
URL (either JDBC or thrift URI) can proxied to Glue Data Catalog, which is usually done within AWS EMR or Glue job environment.
For Hudi streamer, users can set
diff --git a/website/versioned_docs/version-0.15.0/syncing_metastore.md b/website/versioned_docs/version-0.15.0/syncing_metastore.md
index 4006ce8891ed..50a53a70ab14 100644
--- a/website/versioned_docs/version-0.15.0/syncing_metastore.md
+++ b/website/versioned_docs/version-0.15.0/syncing_metastore.md
@@ -124,7 +124,7 @@ will be inferred and set automatically.
## Hive Sync Tool
-Writing data with [DataSource](writing_data) writer or [HoodieStreamer](hoodie_streaming_ingestion) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
+Writing data with [DataSource](writing_data.md) writer or [HoodieStreamer](hoodie_streaming_ingestion.md) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
In case, it's preferable to run this from commandline or in an independent jvm, Hudi provides a `HiveSyncTool`, which can be invoked as below,
once you have built the hudi-hive module. Following is how we sync the above Datasource Writer written table to Hive metastore.
diff --git a/website/versioned_docs/version-0.15.0/use_cases.md b/website/versioned_docs/version-0.15.0/use_cases.md
index fb6061b8d1b5..61ea8ccddb8a 100644
--- a/website/versioned_docs/version-0.15.0/use_cases.md
+++ b/website/versioned_docs/version-0.15.0/use_cases.md
@@ -85,7 +85,7 @@ stream processing world to ensure pipelines don't break from non backwards compa
### ACID Transactions
Along with a table, Apache Hudi brings ACID transactional guarantees to a data lake.
-Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline), stamped with an
+Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline.md), stamped with an
instant time that denotes the time at which the action
is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between writer processes
(that issue userβs upserts/deletes), table services (that write data/metadata to optimize/perform bookkeeping) and readers
@@ -127,12 +127,12 @@ cost savings for your data lake.
Some examples of the Apache Hudi services that make this performance optimization easy include:
-- [Auto File Sizing](file_sizing) - to solve the "small files" problem.
-- [Clustering](clustering) - to co-locate data next to each other.
-- [Compaction](compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](indexing) - for efficient upserts and deletes.
+- [Auto File Sizing](file_sizing.md) - to solve the "small files" problem.
+- [Clustering](clustering.md) - to co-locate data next to each other.
+- [Compaction](compaction.md) - to allow tuning of low latency ingestion and fast read queries.
+- [Indexing](indexing.md) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner.md) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.15.0/write_operations.md b/website/versioned_docs/version-0.15.0/write_operations.md
index bfd007886ab4..c1642f45d1c0 100644
--- a/website/versioned_docs/version-0.15.0/write_operations.md
+++ b/website/versioned_docs/version-0.15.0/write_operations.md
@@ -85,27 +85,27 @@ Here are the basic configs relevant to the write operations types mentioned abov
## Writing path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](configurations#hoodiecombinebeforeinsert)
+1. [Deduping](configurations.md#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](indexing)
+2. [Index Lookup](indexing.md)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](file_sizing)
+3. [File Sizing](file_sizing.md)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](file_layouts)
+4. [Partitioning](file_layouts.md)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](indexing)
+6. Update [Index](indexing.md)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback) can be configured.)
-8. [Clean](hoodie_cleaner) (if needed)
+ 1. Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback.md) can be configured.)
+8. [Clean](hoodie_cleaner.md) (if needed)
1. Following the commit, cleaning is invoked if needed.
-9. [Compaction](compaction)
+9. [Compaction](compaction.md)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
10. Archive
- 1. Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
+ 1. Lastly, we perform an archival step which moves old [timeline](timeline.md) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-0.15.0/writing_data.md b/website/versioned_docs/version-0.15.0/writing_data.md
index bbb2f724c5d1..fd1102fe81c1 100644
--- a/website/versioned_docs/version-0.15.0/writing_data.md
+++ b/website/versioned_docs/version-0.15.0/writing_data.md
@@ -28,15 +28,15 @@ Default value: `"partitionpath"`<br/>
**PRECOMBINE_FIELD**: When two records within the same batch have the same key value, the record with the largest value from the field specified will be choosen. If you are using default payload of OverwriteWithLatestAvroPayload for HoodieRecordPayload (`WRITE_PAYLOAD_CLASS`), an incoming record will always takes precendence compared to the one in storage ignoring this `PRECOMBINE_FIELD_OPT_KEY`. <br/>
Default value: `"ts"`<br/>
-**OPERATION**: The [write operations](write_operations) to use.<br/>
+**OPERATION**: The [write operations](write_operations.md) to use.<br/>
Available values:<br/>
`"upsert"` (default), `"bulk_insert"`, `"insert"`, `"delete"`
-**TABLE_TYPE**: The [type of table](concepts#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
+**TABLE_TYPE**: The [type of table](concepts.md#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
Available values:<br/>
-[`COW_TABLE_TYPE_OPT_VAL`](concepts#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts#merge-on-read-table)
+[`COW_TABLE_TYPE_OPT_VAL`](concepts.md#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts.md#merge-on-read-table)
-**KEYGENERATOR_CLASS_NAME**: Refer to [Key Generation](key_generation) section below.
+**KEYGENERATOR_CLASS_NAME**: Refer to [Key Generation](key_generation.md) section below.
Example:
@@ -84,9 +84,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -120,9 +120,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -422,7 +422,7 @@ Then any record you want to delete you can mark `_hoodie_is_deleted` as true:
Following is an example of how to use `optimistic_concurrency_control` via Spark DataSource API.
-Read more in-depth details about concurrency control in the [concurrency control concepts](concurrency_control) section.
+Read more in-depth details about concurrency control in the [concurrency control concepts](concurrency_control.md) section.
```java
inputDF.write.format("hudi")
diff --git a/website/versioned_docs/version-1.0.0/basic_configurations.md b/website/versioned_docs/version-1.0.0/basic_configurations.md
index 65f404721017..c823c2522f44 100644
--- a/website/versioned_docs/version-1.0.0/basic_configurations.md
+++ b/website/versioned_docs/version-1.0.0/basic_configurations.md
@@ -1,11 +1,11 @@
---
title: Basic Configurations
-summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
last_modified_at: 2024-12-06T17:38:05.854
---
-This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
diff --git a/website/versioned_docs/version-1.0.0/cleaning.md b/website/versioned_docs/version-1.0.0/cleaning.md
index 58986bc10d94..0e004dd1997d 100644
--- a/website/versioned_docs/version-1.0.0/cleaning.md
+++ b/website/versioned_docs/version-1.0.0/cleaning.md
@@ -133,7 +133,7 @@ spark-submit --master local \
Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
#### CLI
-You can also use [Hudi CLI](cli) to run Hoodie Cleaner.
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
CLI provides the below commands for cleaner service:
- `cleans show`
diff --git a/website/versioned_docs/version-1.0.0/cli.md b/website/versioned_docs/version-1.0.0/cli.md
index 7133a42cf14b..5030d17ac5b8 100644
--- a/website/versioned_docs/version-1.0.0/cli.md
+++ b/website/versioned_docs/version-1.0.0/cli.md
@@ -577,7 +577,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-1.0.0/cloud.md b/website/versioned_docs/version-1.0.0/cloud.md
index 123abd5e6bea..a3477b551bee 100644
--- a/website/versioned_docs/version-1.0.0/cloud.md
+++ b/website/versioned_docs/version-1.0.0/cloud.md
@@ -11,23 +11,23 @@ last_modified_at: 2021-10-12T10:50:00+08:00
Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
to cloud stores.
-* [AWS S3](s3_hoodie) <br/>
+* [AWS S3](s3_hoodie.md) <br/>
Configurations required for S3 and Hudi co-operability.
-* [Google Cloud Storage](gcs_hoodie) <br/>
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
Configurations required for GCS and Hudi co-operability.
-* [Alibaba Cloud OSS](oss_hoodie) <br/>
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
Configurations required for OSS and Hudi co-operability.
-* [Microsoft Azure](azure_hoodie) <br/>
+* [Microsoft Azure](azure_hoodie.md) <br/>
Configurations required for Azure and Hudi co-operability.
-* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
Configurations required for COS and Hudi co-operability.
-* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
Configurations required for IBM Cloud Object Storage and Hudi co-operability.
-* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
Configurations required for BOS and Hudi co-operability.
-* [JuiceFS](jfs_hoodie) <br/>
+* [JuiceFS](jfs_hoodie.md) <br/>
Configurations required for JuiceFS and Hudi co-operability.
-* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
Configurations required for OCI and Hudi co-operability.
:::note
diff --git a/website/versioned_docs/version-1.0.0/clustering.md b/website/versioned_docs/version-1.0.0/clustering.md
index e7d160c8acb0..c44bf8d4100a 100644
--- a/website/versioned_docs/version-1.0.0/clustering.md
+++ b/website/versioned_docs/version-1.0.0/clustering.md
@@ -13,7 +13,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## How is compaction different from clustering?
Hudi is modeled like a log-structured storage engine with multiple versions of the data.
-Particularly, [Merge-On-Read](table_types#merge-on-read-table)
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
@@ -93,7 +93,7 @@ update strategy.
This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
-config [hoodie.clustering.plan.strategy.class](configurations#hoodieclusteringplanstrategyclass).
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
Different plan strategies are as follows:
diff --git a/website/versioned_docs/version-1.0.0/compaction.md b/website/versioned_docs/version-1.0.0/compaction.md
index fecfa0834553..c2ca0528551c 100644
--- a/website/versioned_docs/version-1.0.0/compaction.md
+++ b/website/versioned_docs/version-1.0.0/compaction.md
@@ -13,7 +13,7 @@ not applicable to Copy On Write(COW) tables and only applies to MOR tables.
### Why MOR tables need compaction?
To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
-data is organized in terms of [file groups](storage_layouts). Each file group in a MOR table
+data is organized in terms of [file groups](storage_layouts.md). Each file group in a MOR table
consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
are appended to log files.
@@ -182,7 +182,7 @@ This model may need a lock provider **if** metadata table is enabled.
:::
#### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
The compactor utility allows to do scheduling and execution of compaction.
Example:
@@ -199,7 +199,7 @@ Note, the `instant-time` parameter is now optional for the Hudi Compactor Utilit
the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
#### Hudi CLI
-Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-1.0.0/concurrency_control.md b/website/versioned_docs/version-1.0.0/concurrency_control.md
index 549f1ddd17eb..5f5ffbc8164a 100644
--- a/website/versioned_docs/version-1.0.0/concurrency_control.md
+++ b/website/versioned_docs/version-1.0.0/concurrency_control.md
@@ -8,7 +8,7 @@ last_modified_at: 2021-03-19T15:59:57-04:00
---
Concurrency control defines how different writers/readers/table services coordinate access to a Hudi table. Hudi ensures atomic writes, by way of publishing commits atomically to the timeline,
stamped with an instant time that denotes the time at which the action is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between
-writer processes that issue [write operations](write_operations) and table services that (re)write data/metadata to optimize/perform bookkeeping and
+writer processes that issue [write operations](write_operations.md) and table services that (re)write data/metadata to optimize/perform bookkeeping and
readers (that execute queries and read data).
Hudi provides
@@ -31,7 +31,7 @@ hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLock
## Distributed Locking
A pre-requisite for distributed co-ordination in Hudi, like many other distributed database systems is a distributed lock provider, that different processes can use to plan, schedule and
-execute actions on the Hudi timeline in a concurrent fashion. Locks are also used to [generate TrueTime](timeline#truetime-generation), as discussed before.
+execute actions on the Hudi timeline in a concurrent fashion. Locks are also used to [generate TrueTime](timeline.md#truetime-generation), as discussed before.
External locking is typically used in conjunction with optimistic concurrency control
because it provides a way to prevent conflicts that might occur when two or more transactions (commits in our case) attempt to modify the same resource concurrently.
@@ -212,7 +212,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](sql_dml#non-blocking-concurrency-control-experimental).
+writers [here](sql_dml.md#non-blocking-concurrency-control-experimental).
:::note
`NON_BLOCKING_CONCURRENCY_CONTROL` between ingestion writer and table service writer is not yet supported for clustering.
diff --git a/website/versioned_docs/version-1.0.0/deployment.md b/website/versioned_docs/version-1.0.0/deployment.md
index ef23503e2381..19887fab39a1 100644
--- a/website/versioned_docs/version-1.0.0/deployment.md
+++ b/website/versioned_docs/version-1.0.0/deployment.md
@@ -26,13 +26,13 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### Hudi Streamer
-[Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
To use Hudi Streamer in Spark, the `hudi-utilities-slim-bundle` and Hudi Spark bundle are required, by adding
`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.0` to the `spark-submit` command.
- - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -133,7 +133,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Batch Writes](writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.d [...]
+As described in [Batch Writes](writing_data.md#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
@@ -260,7 +260,7 @@ hoodie.table.version=5
```
Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
-For more details on downgrade, please refer documentation [here](cli#upgrade-and-downgrade-table).
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
```shell
connect --path /tmp/hudi_table
downgrade table --toVersion 4
@@ -281,4 +281,4 @@ Currently migrating to Hudi can be done using two approaches
- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
-For more details, refer to the detailed [migration guide](migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-1.0.0/faq.md b/website/versioned_docs/version-1.0.0/faq.md
index 26c3eb50d214..c53c8ebbc68c 100644
--- a/website/versioned_docs/version-1.0.0/faq.md
+++ b/website/versioned_docs/version-1.0.0/faq.md
@@ -6,10 +6,10 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](faq_general)
-- [Design & Concepts](faq_design_and_concepts)
-- [Writing Tables](faq_writing_tables)
-- [Reading Tables](faq_reading_tables)
-- [Table Services](faq_table_services)
-- [Storage](faq_storage)
-- [Integrations](faq_integrations)
+- [General](faq_general.md)
+- [Design & Concepts](faq_design_and_concepts.md)
+- [Writing Tables](faq_writing_tables.md)
+- [Reading Tables](faq_reading_tables.md)
+- [Table Services](faq_table_services.md)
+- [Storage](faq_storage.md)
+- [Integrations](faq_integrations.md)
diff --git a/website/versioned_docs/version-1.0.0/faq_design_and_concepts.md b/website/versioned_docs/version-1.0.0/faq_design_and_concepts.md
index 720f4ae8e2df..8aee10defc1a 100644
--- a/website/versioned_docs/version-1.0.0/faq_design_and_concepts.md
+++ b/website/versioned_docs/version-1.0.0/faq_design_and_concepts.md
@@ -6,14 +6,14 @@ keywords: [hudi, writing, reading]
### How does Hudi ensure atomicity?
-Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
+Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline.md) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
### Does Hudi extend the Hive table layout?
Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
-* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
-* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users exper [...]
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
+* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering.md) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users ex [...]
* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudiβs design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
### What concurrency control approaches does Hudi adopt?
@@ -32,7 +32,7 @@ In these scenarios, it might be tempting to think of data inconsistencies/data l
(examples [1](https://github.com/apache/hudi/blob/aea5bb6f0ab824247f5e3498762ad94f643a2cb6/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L76),
[2](https://github.com/apache/hudi/blame/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L173)) in incremental queries to ensure that no data
is served beyond the point there is an inflight instant in its timeline, so no data loss or drop happens. This detection is made possible because Hudi writes first request a transaction on the timeline, before planning/executing
-the write, as explained in the [timeline](timeline#states) section.
+the write, as explained in the [timeline](timeline.md#states) section.
In this case, on seeing C1βs inflight commit (publish to timeline is atomic), C2 data (which is > C1 in the timeline) is not served until C1 inflight transitions to a terminal state such as completed or marked as failed.
This [test](https://github.com/apache/hudi/blob/master/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java#L137) demonstrates how Hudi incremental source stops proceeding until C1 completes.
@@ -48,7 +48,7 @@ To expand more on the long term approach, Hudi has had a proposal to streamline/
This has been delayed for a few reasons
- Large hosted query engines and users not upgrading fast enough.
-- The issues brought up - \[[1](faq_design_and_concepts#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
+- The issues brought up - \[[1](faq_design_and_concepts.md#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts.md#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
relevant to this are not practically very important to users beyond good pedantic discussions,
- Wanting to do it alongside [non-blocking concurrency control](https://github.com/apache/hudi/pull/7907) in Hudi version 1.x.
diff --git a/website/versioned_docs/version-1.0.0/faq_general.md b/website/versioned_docs/version-1.0.0/faq_general.md
index cb9e0161bcb8..bfefd3b6d851 100644
--- a/website/versioned_docs/version-1.0.0/faq_general.md
+++ b/website/versioned_docs/version-1.0.0/faq_general.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### When is Hudi useful for me or my organization?
-If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
+If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion.md). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
@@ -37,7 +37,7 @@ Choose Copy-on-write storage if :
* You are looking for a simple alternative, that replaces your existing parquet tables without any need for real-time data.
* Your current job is rewriting entire table/partition to deal with updates, while only a few files actually change in each partition.
-* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
+* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations.md#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
* Your workload is fairly well-understood and does not have sudden bursts of large amount of update or inserts to older partitions. COW absorbs all the merging cost on the writer side and thus these sudden changes can clog up your ingestion and interfere with meeting normal mode ingest latency targets.
Choose merge-on-read storage if :
@@ -51,7 +51,7 @@ Immaterial of what you choose, Hudi provides
* Incremental pulls
* Ability to de-duplicate data
-Find more [here](concepts/).
+Find more [here](concepts.md).
### Is Hudi an analytical database?
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data.md) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
@@ -79,7 +79,7 @@ generate the key internally (much like RDBMS row_ids)
### How does Hudi actually store data inside a table?
-At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts/) for more details.
+At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts.md) for more details.
### How Hudi handles partition evolution requirements ?
diff --git a/website/versioned_docs/version-1.0.0/faq_reading_tables.md b/website/versioned_docs/version-1.0.0/faq_reading_tables.md
index 3f413b7ef1e0..e9db97dbbdd2 100644
--- a/website/versioned_docs/version-1.0.0/faq_reading_tables.md
+++ b/website/versioned_docs/version-1.0.0/faq_reading_tables.md
@@ -24,7 +24,7 @@ set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
### Does Hudi guarantee consistent reads? How to think about read optimized queries?
-Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
+Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types.md#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
Hudiβs read-optimized query is targeted for the MOR table only, with guidance around how compaction should be run to achieve predictable results. In the MOR table, the compaction, which runs every few commits (or βdeltacommitβ to be exact for the MOR table) by default, merges the base (parquet) file and corresponding change log files to a new base file within each file group, so that the snapshot query serving the latest data immediately after compaction reads the base files only.Β Simil [...]
diff --git a/website/versioned_docs/version-1.0.0/faq_storage.md b/website/versioned_docs/version-1.0.0/faq_storage.md
index b088b30d8ebd..2f6950fed334 100644
--- a/website/versioned_docs/version-1.0.0/faq_storage.md
+++ b/website/versioned_docs/version-1.0.0/faq_storage.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### Does Hudi support cloud storage/object stores?
-Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
+Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud.md) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations.md#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
### What is the difference between copy-on-write (COW) vs merge-on-read (MOR) table types?
@@ -14,11 +14,11 @@ Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop
**Merge On Read** - This storage type enables clients to ingest data quickly onto row based data format such as avro. Any new data that is written to the Hudi table using MOR table type, will write new log/delta files that internally store the data as avro encoded bytes. A compaction process (configured as inline or asynchronous) will convert log file format to columnar file format (parquet). Two different InputFormats expose 2 different views of this data, Read Optimized view exposes co [...]
-More details can be found [here](concepts/) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
+More details can be found [here](concepts.md) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
### How do I migrate my data to Hudi?
-Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide/), including ways to doing partial migrations.
+Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables.md#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide.md), including ways to doing partial migrations.
### How to convert an existing COW table to MOR?
@@ -34,7 +34,7 @@ But manually changing it will result in checksum errors. So, we have to go via h
### How can I find the average record size in a commit?
-The `commit showpartitons` command in [HUDI CLI](cli) will show both "bytes written" and
+The `commit showpartitons` command in [HUDI CLI](cli.md) will show both "bytes written" and
"records inserted." Divide the bytes written by records inserted to find the average size. Note that this answer assumes
@@ -91,11 +91,11 @@ spark.read.parquet("your_data_set/path/to/month").limit(n) // Limit n records
.save(basePath);
```
-For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli).
+For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli.md).
### Why does maintain record level commit metadata? Isn't tracking table version at file level good enough?Β
-By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hudi [...]
+By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering.md#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hu [...]
### Why partition fields are also stored in parquet files in addition to the partition path ?
diff --git a/website/versioned_docs/version-1.0.0/faq_table_services.md b/website/versioned_docs/version-1.0.0/faq_table_services.md
index 7af1ec75398b..dc4dbcf9716f 100644
--- a/website/versioned_docs/version-1.0.0/faq_table_services.md
+++ b/website/versioned_docs/version-1.0.0/faq_table_services.md
@@ -6,11 +6,11 @@ keywords: [hudi, writing, reading]
### What does the Hudi cleaner do?
-The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be read [...]
+The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations.md#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be r [...]
### How do I run compaction for a MOR table?
-Simplest way to run compaction on MOR table is to run the [compaction inline](configurations#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted most o [...]
+Simplest way to run compaction on MOR table is to run the [compaction inline](configurations.md#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted mos [...]
That said, for obvious reasons of not blocking ingesting for compaction, you may want to run it asynchronously as well. This can be done either via a separate [compaction job](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java) that is scheduled by your workflow scheduler/notebook independently. If you are using Hudi Streamer, then you can run in [continuous mode](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca [...]
@@ -40,7 +40,7 @@ Depending on how you write to Hudi these are the possible options currently.
### How to disable all table services in case of multiple writers?
-[hoodie.table.services.enabled](configurations#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
+[hoodie.table.services.enabled](configurations.md#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
### Why does Hudi retain at-least one previous commit even after setting hoodie.cleaner.commits.retained': 1 ?
@@ -50,6 +50,6 @@ Hudi runs cleaner to remove old file versions as part of writing data either in
Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
-be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback)
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback.md)
for details
diff --git a/website/versioned_docs/version-1.0.0/faq_writing_tables.md b/website/versioned_docs/version-1.0.0/faq_writing_tables.md
index fe061d1b491a..94badc5daa81 100644
--- a/website/versioned_docs/version-1.0.0/faq_writing_tables.md
+++ b/website/versioned_docs/version-1.0.0/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API and us [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations.md) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API a [...]
### How is a Hudi writer job deployed?
@@ -68,21 +68,21 @@ As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github
### How do I delete records in the dataset using Hudi?
-GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data#deletes).
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data.md#deletes).
### Should I need to worry about deleting all copies of the records in case of duplicates?
-No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https://githu [...]
+No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables.md#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables.md#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https:/ [...]
### How does Hudi handle duplicate record keys in an input?
-When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
+When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
-For an insert or bulk_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in [...]
+For an insert or bulk_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or [...]
### How can I pass hudi configurations to my spark writer job?
-Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations/). Invoking _--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
+Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations.md). Invoking _--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
* For Spark DataSource, you can use the "options" API of DataFrameWriter to pass in these configs.
@@ -108,28 +108,28 @@ hoodie.datasource.write.hive_style_partitioning: true
### Can I register my Hudi table with Apache Hive metastore?
-Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations#hoodiedatasourcehive_syncenable).
+Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore.md#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations.md#hoodiedatasourcehive_syncenable).
### What's Hudi's schema evolution story?
Hudi uses Avro as the internal canonical representation for records, primarily due to its nice [schema compatibility & evolution](https://docs.confluent.io/platform/current/schema-registry/avro.html) properties. This is a key aspect of having reliability in your ingestion or ETL pipelines. As long as the schema passed to Hudi (either explicitly in Hudi Streamer schema provider configs or implicitly by Spark Datasource's Dataset schemas) is backwards compatible (e.g no field deletes, only [...]
-Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution) for more details on Schema-on-read for Spark..
+Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution.md) for more details on Schema-on-read for Spark..
### What performance/ingest latency can I expect for Hudi writing?
-The speed at which you can write into Hudi depends on the [write operation](write_operations) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when try [...]
+The speed at which you can write into Hudi depends on the [write operation](write_operations.md) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when [...]
| Storage Type | Type of workload | Performance | Tips |
| ---| ---| ---| --- |
-| copy on write | bulk_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism) to get right number of files. Use insert if you want this auto tuned. Configure [hoodie.bulkinsert.sort.mode](configurations#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value `NONE` offers the fastest performance and matches `spark.write.parquet()` [...]
-| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
-| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or less than 50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For example, if a partitio [...]
+| copy on write | bulk_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism) to get right number of files. Use insert if you want this auto tuned. Configure [hoodie.bulkinsert.sort.mode](configurations.md#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value `NONE` offers the fastest performance and matches `spark.write.parqu [...]
+| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations.md#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
+| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or less than 50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For example, if a partitio [...]
| merge on read | bulk insert | Currently new data only goes to parquet files and thus performance here should be similar to copy on write bulk insert. This has the nice side-effect of getting data into parquet directly for query performance. [This GitHub issue](https://github.com/apache/hudi/issues/14468) will add support for logging inserts directly and this up drastically. | |
| merge on read | insert | Similar to above | |
| merge on read | upsert/ de-duplicate & insert | Indexing performance would remain the same as copy-on-write, while ingest latency for updates (costliest I/O operation in copy on write) are sent to log files and thus with asynchronous compaction provides very good ingest performance with low write amplification. | |
-Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide) for more tips on JVM and other configurations.
+Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide.md) for more tips on JVM and other configurations.
### What performance can I expect for Hudi reading/queries?
@@ -147,11 +147,11 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations#hoodieparquetmaxfilesize) and the [soft limit](configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on previou [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations.md#hoodieparquetmaxfilesize) and the [soft limit](configurations.md#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on p [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
-* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations#hoodielogfilemaxsize) and a [factor](configurations#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
+* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations.md#hoodielogfilemaxsize) and a [factor](configurations.md#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
* Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations as above for the COPY_ON_WRITE case applies.
NOTE : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the compaction has been p [...]
@@ -173,7 +173,7 @@ hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPa
With each commit, Hudi creates a new table version in the metastore. This can be reduced by setting the option
-[hoodie.datasource.meta_sync.condition.sync](configurations#hoodiedatasourcemeta_syncconditionsync) to true.
+[hoodie.datasource.meta_sync.condition.sync](configurations.md#hoodiedatasourcemeta_syncconditionsync) to true.
This will ensure that hive sync is triggered on schema or partitions changes.
@@ -183,11 +183,11 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
+Hudi employs [optimistic concurrency control](concurrency_control.md) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/ [...]
### Can single-writer inserts have duplicates?
-By default, Hudi turns off key based de-duplication for INSERT/BULK_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in delta [...]
+By default, Hudi turns off key based de-duplication for INSERT/BULK_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in de [...]
### Can concurrent inserts cause duplicates?
diff --git a/website/versioned_docs/version-1.0.0/file_sizing.md b/website/versioned_docs/version-1.0.0/file_sizing.md
index 62ad0f7a4320..02ec64a8fe43 100644
--- a/website/versioned_docs/version-1.0.0/file_sizing.md
+++ b/website/versioned_docs/version-1.0.0/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning.md)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-1.0.0/flink-quick-start-guide.md b/website/versioned_docs/version-1.0.0/flink-quick-start-guide.md
index 125ac059c2be..07f2e280ebd1 100644
--- a/website/versioned_docs/version-1.0.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.0/flink-quick-start-guide.md
@@ -60,7 +60,7 @@ quick start tool for SQL users.
### Start Flink SQL client
Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
-You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore#install)), or download it from the
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
Now start the SQL CLI:
@@ -282,7 +282,7 @@ Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-e
>
This statement queries snapshot view of the dataset.
-Refers to [Table types and queries](concepts#table-types--queries) for more info on all table types and query types supported.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
## Update Data
@@ -325,7 +325,7 @@ Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/ma
</Tabs
>
-[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts)
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
denoted by the timestamp.
@@ -449,19 +449,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink#cdc-ingestion), [Bulk Insert](ingestion_flink#bulk-insert), [Index Bootstrap](ingestion_flink#index-bootstrap), [Changelog Mode](ingestion_flink#changelog-mode) and [Append Mode](ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml#non-blocking-concurrency-control-experimental).
-- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](sql_queries#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
-- **Optimization**: Offline compaction is supported [Offline Compaction](compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk Insert](ingestion_flink.md#bulk-insert), [Index Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog Mode](ingestion_flink.md#changelog-mode) and [Append Mode](ingestion_flink.md#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml.md#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](sql_queries.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
- - [Hudi Storage Layout](storage_layouts) - How the files are laid out on storage
- - [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+ - [Hudi Storage Layout](storage_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -470,5 +470,5 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it. Also, if you are looking for ways to migrate your existing data
-to Hudi, refer to [migration guide](migration_guide).
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-1.0.0/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.0/hoodie_streaming_ingestion.md
index 5ad0494de41f..ceb0313a4a25 100644
--- a/website/versioned_docs/version-1.0.0/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.0.0/hoodie_streaming_ingestion.md
@@ -268,7 +268,7 @@ and then ingest it as follows.
--op BULK_INSERT
```
-In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide).
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
### Using `hudi-utilities-slim-bundle` bundle jar
@@ -303,7 +303,7 @@ A Hudi Streamer job can then be triggered as follows:
--op BULK_INSERT
```
-Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control) section
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
### Checkpointing
@@ -314,7 +314,7 @@ Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single [...]
+- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a sing [...]
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
diff --git a/website/versioned_docs/version-1.0.0/hudi_stack.md b/website/versioned_docs/version-1.0.0/hudi_stack.md
index 214a1ba2f7aa..bd5d3e5cf152 100644
--- a/website/versioned_docs/version-1.0.0/hudi_stack.md
+++ b/website/versioned_docs/version-1.0.0/hudi_stack.md
@@ -49,19 +49,19 @@ bring any compute engine for specific workloads.
Drawing an analogy to file formats, a table format simply concerns with how files are distributed with the table, partitioning schemes, schema and metadata tracking changes. Hudi organizes files within a table or partition into
File Groups. Updates are captured in log files tied to these File Groups, ensuring efficient merges. There are three major components related to Hudiβs table format.
-- **Timeline** : Hudi's [timeline](timeline), stored in the `/.hoodie/timeline` folder, is a crucial event log recording all table actions in an ordered manner,
+- **Timeline** : Hudi's [timeline](timeline.md), stored in the `/.hoodie/timeline` folder, is a crucial event log recording all table actions in an ordered manner,
with events kept for a specified period. Hudi uniquely designs each File Group as a self-contained log, enabling record state reconstruction through delta logs, even after archival of historical actions. This approach effectively limits metadata size based on table activity frequency, essential for managing tables with frequent updates.
- **File Group and File Slice** : Within each partition the data is physically stored as base and Log Files and organized into logical concepts as [File groups](https://hudi.apache.org/tech-specs-1point0/#storage-layout) and
File Slices. File groups contain multiple versions of File Slices and are split into multiple File Slices. A File Slice comprises the Base and Log File. Each File Slice within
the file-group is uniquely identified by the write that created its base file or the first log file, which helps order the File Slices.
-- **Metadata Table** : Implemented as another merge-on-read Hudi table, the [metadata table](metadata) efficiently handles quick updates with low write amplification.
+- **Metadata Table** : Implemented as another merge-on-read Hudi table, the [metadata table](metadata.md) efficiently handles quick updates with low write amplification.
It leverages a [SSTable](https://cassandra.apache.org/doc/stable/cassandra/architecture/storage-engine.html#sstables) based file format for quick, indexed key lookups,
storing vital information like file paths, column statistics and schema. This approach streamlines operations by reducing the necessity for expensive cloud file listings.
Hudiβs approach of recording updates into Log Files is more efficient and involves low merge overhead than systems like Hive ACID, where merging all delta records against
-all Base Files is required. Read more about the various table types in Hudi [here](table_types).
+all Base Files is required. Read more about the various table types in Hudi [here](table_types.md).
## Storage Engine
@@ -74,8 +74,8 @@ Cassandra and Clickhouse.
<p align = "center">Figure: Indexes in Hudi</p>
-[Indexes](indexes) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. The [metadata table](metadata/#metadata-table-indices) acts
-as an additional [indexing system](metadata#supporting-multi-modal-index-in-hudi) and brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata
+[Indexes](indexes.md) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. The [metadata table](metadata/#metadata-table-indices) acts
+as an additional [indexing system](metadata.md#supporting-multi-modal-index-in-hudi) and brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata
table, like file listings, column statistics, bloom filters, record-level indexes, and [expression indexes](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) to quickly generate optimized query plans and improve read
performance. In addition to the metadata table indexes, Hudi supports simple join based indexing, bloom filters stored in base file footers, external key-value stores like HBase,
and optimized storage techniques like bucketing , to efficiently locate File Groups containing specific record keys. Hudi also provides reader indexes such as [expression](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) and
@@ -91,12 +91,12 @@ running them in inline, semi-asynchronous or full-asynchronous modes. Furthermor
asynchronously sharing the underlying executors intelligently with writers. Letβs take a look at these services.
#### Clustering
-The [clustering](clustering) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into
+The [clustering](clustering.md) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into
larger ones for optimal file size management. It's fully integrated with other timeline actions like cleaning and compaction, enabling smart optimizations such as avoiding
compaction for File Groups undergoing clustering, thereby saving on I/O.
#### Compaction
-Hudi's [compaction](compaction) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows
+Hudi's [compaction](compaction.md) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows
concurrent writes to the same File Froup, enabled by Hudi's file grouping and flexible log merging. This facilitates non-blocking execution of deletes even during concurrent
record updates.
@@ -107,11 +107,11 @@ while also allowing sufficient time for long running batch jobs (e.g Hive ETLs)
#### Indexing
Hudi's scalable metadata table contains auxiliary data about the table. This subsystem encompasses various indices, including files, column_stats, and bloom_filters,
facilitating efficient record location and data skipping. Balancing write throughput with index updates presents a fundamental challenge, as traditional indexing methods,
-like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchronous [metadata indexing](metadata_indexing),
+like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchronous [metadata indexing](metadata_indexing.md),
enabling the creation of various indices without impeding writes. This approach not only improves write latency but also minimizes resource waste by reducing contention between writing and indexing activities.
### Concurrency Control
-[Concurrency control](concurrency_control) defines how different writers/readers/table services coordinate access to the table. Hudi uses monotonically increasing time to sequence and order various
+[Concurrency control](concurrency_control.md) defines how different writers/readers/table services coordinate access to the table. Hudi uses monotonically increasing time to sequence and order various
changes to table state. Much like databases, Hudi take an approach of clearly differentiating between writers (responsible for upserts/deletes), table services
(focusing on storage optimization and bookkeeping), and readers (for query execution). Hudi provides snapshot isolation, offering a consistent view of the table across
these different operations. It employs lock-free, non-blocking MVCC for concurrency between writers and table-services, as well as between different table services, and
@@ -154,12 +154,12 @@ integration with engines written in C/C++.
<p align = "center">Figure: Various platform services in Hudi</p>
Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers.
-Services, like [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
+Services, like [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (including Confluent), and
deduplication of data. Hudi Streamer also offers features for backfills, one-off runs, and continuous mode operation with Spark/Flink streaming writers. Additionally,
-Hudi provides tools for [snapshotting](snapshot_exporter) and incrementally [exporting](snapshot_exporter#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback) for analytics or
+Hudi provides tools for [snapshotting](snapshot_exporter.md) and incrementally [exporting](snapshot_exporter.md#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback.md) for analytics or
workflow management, enhancing the deployment of production-grade incremental pipelines. Apart from these services, Hudi also provides broad support for different
-catalogs such as [Hive Metastore](syncing_metastore), [AWS Glue](syncing_aws_glue_data_catalog/), [Google BigQuery](gcp_bigquery), [DataHub](syncing_datahub), etc. that allows syncing of Hudi tables to be queried by
+catalogs such as [Hive Metastore](syncing_metastore.md), [AWS Glue](syncing_aws_glue_data_catalog.md), [Google BigQuery](gcp_bigquery.md), [DataHub](syncing_datahub.md), etc. that allows syncing of Hudi tables to be queried by
interactive engines such as Trino and Presto.
### Metaserver*
diff --git a/website/versioned_docs/version-1.0.0/indexes.md b/website/versioned_docs/version-1.0.0/indexes.md
index c7c104764acc..08162ef3066f 100644
--- a/website/versioned_docs/version-1.0.0/indexes.md
+++ b/website/versioned_docs/version-1.0.0/indexes.md
@@ -19,8 +19,8 @@ Only clustering or cross-partition updates that are implemented as deletes + ins
file group at any completed instant on the timeline.
## Need for indexing
-For [Copy-On-Write tables](table_types#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](table_types#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
only against updates for records that are part of that base file.
@@ -28,7 +28,7 @@ only against updates for records that are part of that base file.
In contrast,
- Designs without an indexing component (e.g: [Apache Hive/Apache Iceberg](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)) end up having to merge all the base files against all incoming updates/delete records
- (10-100x more [read amplification](table_types#comparison)).
+ (10-100x more [read amplification](table_types.md#comparison)).
- Designs that implement heavily write-optimized OLTP data structures like LSM trees do not require an indexing component. But they perform poorly scan heavy workloads
against cloud storage making them unsuitable for serving analytical queries.
@@ -42,8 +42,8 @@ implemented by enhancing the metadata table with the flexibility to extend to ne
along with an [asynchronous index](https://hudi.apache.org/docs/metadata_indexing/#setup-async-indexing) building
Hudi supports a multi-modal index by augmenting the metadata table with the capability to incorporate new types of indexes, complemented by an
-asynchronous mechanism for [index construction](metadata_indexing). This enhancement supports a range of indexes within
-the [metadata table](metadata#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
+asynchronous mechanism for [index construction](metadata_indexing.md). This enhancement supports a range of indexes within
+the [metadata table](metadata.md#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
<p align = "center">Figure: Indexes in Hudi</p>
@@ -90,7 +90,7 @@ Following are configurations that control enabling record index building and mai
An [expression index](https://github.com/apache/hudi/blob/3789840be3d041cbcfc6b24786740210e4e6d6ac/rfc/rfc-63/rfc-63.md) is an index on a function of a column. If a query has a predicate on a function of a column, the expression index can
be used to speed up the query. Expression index is stored in *expr_index_* prefixed partitions (one for each
expression index) under metadata table. Expression index can be created using SQL syntax. Please checkout SQL DDL
-docs [here](sql_ddl#create-expression-index) for more details.
+docs [here](sql_ddl.md#create-expression-index) for more details.
Following are configurations that control enabling expression index building and maintenance on the writer.
diff --git a/website/versioned_docs/version-1.0.0/ingestion_flink.md b/website/versioned_docs/version-1.0.0/ingestion_flink.md
index 8176ff1e5ac6..8ec300910490 100644
--- a/website/versioned_docs/version-1.0.0/ingestion_flink.md
+++ b/website/versioned_docs/version-1.0.0/ingestion_flink.md
@@ -37,7 +37,7 @@ frequent `file handle` switching.
:::note
The parallelism of `bulk_insert` is specified by `write.tasks`. The parallelism will affect the number of small files.
In theory, the parallelism of `bulk_insert` is the number of `bucket`s (In particular, when each bucket writes to maximum file size, it
-will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks).
+will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks).
:::
#### Options
@@ -45,7 +45,7 @@ will rollover to the new file handle. Finally, `the number of files` >= [`write.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.operation` | `true` | `upsert` | Setting as `bulk_insert` to open this function |
-| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks) |
+| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks) |
| `write.bulk_insert.shuffle_input` | `false` | `true` | Whether to shuffle data according to the input field before writing. Enabling this option will reduce the number of small files, but there may be a risk of data skew |
| `write.bulk_insert.sort_input` | `false` | `true` | Whether to sort data according to the input field before writing. Enabling this option will reduce the number of small files when a write task writes multiple partitions |
| `write.sort.memory` | `false` | `128` | Available managed memory of sort operator. default `128` MB |
diff --git a/website/versioned_docs/version-1.0.0/metadata.md b/website/versioned_docs/version-1.0.0/metadata.md
index 7606dc08bce0..55934e860a27 100644
--- a/website/versioned_docs/version-1.0.0/metadata.md
+++ b/website/versioned_docs/version-1.0.0/metadata.md
@@ -62,12 +62,12 @@ Following are the different types of metadata currently supported.
even without examining column statistics at the file level. The partition stats index is stored in *partition_stats* partition in the metadata table.
Partition stats index can be enabled using the following configs (note it is required to specify the columns for which stats should be aggregated).
-To try out these features, refer to the [SQL guide](sql_ddl#create-partition-stats-index).
+To try out these features, refer to the [SQL guide](sql_ddl.md#create-partition-stats-index).
## Metadata Tracking on Writers
Following are based basic configs that are needed to enable metadata tracking. For advanced configs please refer
-[here](configurations#Metadata-Configs).
+[here](configurations.md#Metadata-Configs).
| Config Name | Default | Description |
|----------------------------------------------|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -118,10 +118,10 @@ corresponding configs across Spark and Flink readers.
To ensure that metadata table stays up to date and table metadata is tracked safely across concurrent write and
table operations, there are some additional considerations. If async table services are enabled for the table (i.e. running a separate compaction (`HoodieCompactor`) or
clustering (`HoodieClusteringJob`) job), even with just a single writer, lock providers
-must be configured. Please refer to [concurrency control](concurrency_control) for more details.
+must be configured. Please refer to [concurrency control](concurrency_control.md) for more details.
Before enabling metadata table for the first time, all writers on the same table must and table services must be stopped.
-If your current deployment model is [multi-writer](concurrency_control#full-on-multi-writer--async-table-services) along with a lock
+If your current deployment model is [multi-writer](concurrency_control.md#full-on-multi-writer--async-table-services) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only a subset of writers or table services is unsafe and can lead to loss of data. So, please ensure you enable
diff --git a/website/versioned_docs/version-1.0.0/metadata_indexing.md b/website/versioned_docs/version-1.0.0/metadata_indexing.md
index 475b68beb9a4..b32df9d1ccdd 100644
--- a/website/versioned_docs/version-1.0.0/metadata_indexing.md
+++ b/website/versioned_docs/version-1.0.0/metadata_indexing.md
@@ -5,7 +5,7 @@ toc: true
last_modified_at:
---
-Hudi maintains a scalable [metadata](metadata) that has some auxiliary data about the table.
+Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data about the table.
The [pluggable indexing subsystem](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi)
of Hudi depends on the metadata table. Different types of index, from `files` index for locating records efficiently
to `column_stats` index for data skipping, are part of the metadata table. A fundamental tradeoff in any data system
@@ -35,7 +35,7 @@ For more information on these indexes please refer [metadata section](metadata/#
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -54,7 +54,7 @@ CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(
CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
```
-For more information on index creation using SQL refer [SQL DDL](sql_ddl#create-index)
+For more information on index creation using SQL refer [SQL DDL](sql_ddl.md#create-index)
## Index Creation Using Datasource
@@ -129,7 +129,7 @@ In the example we will have continuous writing using Hudi Streamer and also crea
in example is done using HoodieIndexer so that schedule and execute phases are clearly visible for indexing. The asynchronous
configurations can be used with Datasource and SQL based configs to create index as well.
-First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) which will continuously write data
+First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) which will continuously write data
from raw parquet to Hudi table. We used the widely available [NY Taxi dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/), whose setup details are as below:
<details>
<summary>Ingestion write config</summary>
@@ -183,7 +183,7 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](indexes#multi-modal-indexing) along with [configs](metadata#metadata-tracking-on-writers)
+metadata table can be explored [here](indexes.md#multi-modal-indexing) along with [configs](metadata.md#metadata-tracking-on-writers)
to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
diff --git a/website/versioned_docs/version-1.0.0/metrics.md b/website/versioned_docs/version-1.0.0/metrics.md
index c6f2833f5ada..c389689716be 100644
--- a/website/versioned_docs/version-1.0.0/metrics.md
+++ b/website/versioned_docs/version-1.0.0/metrics.md
@@ -6,7 +6,7 @@ toc: true
last_modified_at: 2020-06-20T15:59:57-04:00
---
-In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations#METRICS).
+In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations.md#METRICS).
## MetricsReporter
@@ -17,7 +17,7 @@ MetricsReporter provides APIs for reporting `HoodieMetrics` to user-specified ba
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
#### Configurations
-The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Jmx).
+The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Jmx).
```properties
hoodie.metrics.on=true
@@ -37,7 +37,7 @@ As configured above, JmxMetricsReporter will started JMX server on port 4001. We
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send `HoodieMetrics` to it.
#### Configurations
-The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations#Metrics-Configurations-for-Graphite).
+The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Graphite).
```properties
hoodie.metrics.on=true
@@ -58,7 +58,7 @@ DatadogMetricsReporter is an implementation of Datadog reporter.
A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
#### Configurations
-The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Datadog-reporter).
+The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Datadog-reporter).
```properties
hoodie.metrics.on=true
@@ -138,7 +138,7 @@ tuned are in the `HoodieMetricsCloudWatchConfig` class.
Allows users to define a custom metrics reporter.
#### Configurations
-The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations).
+The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations).
```properties
hoodie.metrics.on=true
diff --git a/website/versioned_docs/version-1.0.0/migration_guide.md b/website/versioned_docs/version-1.0.0/migration_guide.md
index 99bafee92a75..64d87622183b 100644
--- a/website/versioned_docs/version-1.0.0/migration_guide.md
+++ b/website/versioned_docs/version-1.0.0/migration_guide.md
@@ -40,7 +40,7 @@ There are a few options when choosing this approach.
#### Using Hudi Streamer
-Use the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool. HoodieStreamer supports bootstrap with
+Use the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool. HoodieStreamer supports bootstrap with
--run-bootstrap command line option. There are two types of bootstrap, METADATA_ONLY and FULL_RECORD. METADATA_ONLY will
generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will
perform a full copy/rewrite of the data as a Hudi table. Additionally, once can choose selective partitions using regex
@@ -89,7 +89,7 @@ Refer to [Bootstrap procedure](https://hudi.apache.org/docs/next/procedures#boot
#### Using Hudi CLI
Write your own custom logic of how to load an existing table into a Hudi managed one. Please read about the RDD API
-[here](quick-start-guide). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
+[here](quick-start-guide.md). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hudi-cli && ./hudi-cli.sh`.
```java
diff --git a/website/versioned_docs/version-1.0.0/overview.mdx b/website/versioned_docs/version-1.0.0/overview.mdx
index 8f4e35c7fde7..fe320b347e16 100644
--- a/website/versioned_docs/version-1.0.0/overview.mdx
+++ b/website/versioned_docs/version-1.0.0/overview.mdx
@@ -17,14 +17,14 @@ Apache Hudi (pronounced "hoodie") pioneered the concept of "[transactional data
the data lakehouse architecture. Today, Hudi has grown into an [open data lakehouse platform](/blog/2021/07/21/streaming-data-lake-platform), with a open table format purpose-built for high performance writes on
incremental data pipelines and fast query performance due to comprehensive table optimizations.
-Hudi brings core database functionality directly to a data lake - [tables](sql_ddl),
-[transactions](timeline), [efficient upserts/deletes](write_operations), [advanced indexes](indexes),
-[ingestion services](hoodie_streaming_ingestion), data [clustering](clustering)/[compaction](compaction) optimizations,
-and [concurrency control](concurrency_control) all while keeping your data in open file formats. Not only is Apache Hudi great for streaming workloads,
-but it also allows you to create efficient incremental batch pipelines. Apache Hudi can easily be used on any [cloud storage platform](cloud).
+Hudi brings core database functionality directly to a data lake - [tables](sql_ddl.md),
+[transactions](timeline.md), [efficient upserts/deletes](write_operations.md), [advanced indexes](indexes.md),
+[ingestion services](hoodie_streaming_ingestion.md), data [clustering](clustering.md)/[compaction](compaction.md) optimizations,
+and [concurrency control](concurrency_control.md) all while keeping your data in open file formats. Not only is Apache Hudi great for streaming workloads,
+but it also allows you to create efficient incremental batch pipelines. Apache Hudi can easily be used on any [cloud storage platform](cloud.md).
Hudiβs advanced performance optimizations, make analytical queries/pipelines faster with any of the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.
-Read the docs for more [use case descriptions](use_cases) and check out [who's using Hudi](/powered-by), to see how some of the
+Read the docs for more [use case descriptions](use_cases.md) and check out [who's using Hudi](/powered-by), to see how some of the
largest data lakes in the world including [Uber](https://www.uber.com/en-IN/blog/uber-big-data-platform/), [Amazon](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/),
[ByteDance](http://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance),
[Robinhood](https://s.apache.org/hudi-robinhood-talk) and more are transforming their production data lakes with Hudi.
@@ -35,10 +35,10 @@ expands the use of Apache Hudi for a diverse range of use cases in the non-JVM e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
-- [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
-- [Hudi File Layout](storage_layouts) - How the files are laid out on storage
-- [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
-- [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+- [Hudi File Layout](storage_layouts.md) - How the files are laid out on storage
+- [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+- [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Design & Concepts" section of the docs.
@@ -48,11 +48,11 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:
-- [Spark Quick Start Guide](quick-start-guide) β if you primarily use Apache Spark
-- [Flink Quick Start Guide](flink-quick-start-guide) β if you primarily use Apache Flink
-- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide) - if you primarily use Python or Rust
+- [Spark Quick Start Guide](quick-start-guide.md) β if you primarily use Apache Spark
+- [Flink Quick Start Guide](flink-quick-start-guide.md) β if you primarily use Apache Flink
+- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide.md) - if you primarily use Python or Rust
-If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo)
+If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo.md)
## Connect With The Community
Apache Hudi is community-focused and community-led and welcomes new-comers with open arms. Leverage the following
diff --git a/website/versioned_docs/version-1.0.0/performance.md b/website/versioned_docs/version-1.0.0/performance.md
index 8eaa2e32c303..219fb5d1ced8 100644
--- a/website/versioned_docs/version-1.0.0/performance.md
+++ b/website/versioned_docs/version-1.0.0/performance.md
@@ -16,12 +16,12 @@ Here are some ways to efficiently manage the storage of your Hudi tables.
- The [small file handling feature](configurations/#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](configurations#hoodieparquetcompressionratio),
+- Cleaner can be [configured](configurations.md#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](configurations.md#hoodieparquetmaxfilesize), [log files](configurations.md#hoodielogfilemaxsize) & expected [compression ratio](configurations.md#hoodieparquetcompressionratio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
-- Intelligently tuning the [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
+- Intelligently tuning the [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be changed without re-clustering the table. Writes will simply expand given file groups with new updates/inserts as explained before.
-- For workloads with heavy updates, the [merge-on-read table](concepts#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
+- For workloads with heavy updates, the [merge-on-read table](concepts.md#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
## Performance Optimizations
@@ -119,11 +119,11 @@ For very large tables (1Tb+, 10s of 1000s of files), Data skipping could
To unlock the power of Data Skipping you will need to
-1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
+1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing.md)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
2. Enable Data Skipping in your queries, using `hoodie.metadata.index.column.stats.enable=true` (to enable Column Stats Index being populated on the write path, disabled by default)
:::note
-If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
+If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing.md) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
:::
To enable Data Skipping in your queries make sure to set following properties to `true` (on the read path):
diff --git a/website/versioned_docs/version-1.0.0/precommit_validator.md b/website/versioned_docs/version-1.0.0/precommit_validator.md
index ada26dd7a4c8..47a8add2696a 100644
--- a/website/versioned_docs/version-1.0.0/precommit_validator.md
+++ b/website/versioned_docs/version-1.0.0/precommit_validator.md
@@ -8,7 +8,7 @@ Data quality refers to the overall accuracy, completeness, consistency, and vali
Hudi offers **Pre-Commit Validators** that allow you to ensure that your data meets certain data quality expectations as you are writing with Hudi Streamer or Spark Datasource writers.
:::note
-Pre-commit validators are skipped when using the [BULK_INSERT](write_operations#bulk_insert) write operation type.
+Pre-commit validators are skipped when using the [BULK_INSERT](write_operations.md#bulk_insert) write operation type.
:::
Multiple class names can be separated by `,` delimiter.
@@ -105,7 +105,7 @@ void validateRecordsBeforeAndAfter(Dataset<Row> before,
```
## Additional Monitoring with Notifications
-Hudi offers a [commit notification service](platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
+Hudi offers a [commit notification service](platform_services_post_commit_callback.md) that can be configured to trigger notifications about write commits.
The commit notification service can be combined with pre-commit validators to send a notification when a commit fails a validation. This is possible by passing details about the validation as a custom value to the HTTP endpoint.
diff --git a/website/versioned_docs/version-1.0.0/procedures.md b/website/versioned_docs/version-1.0.0/procedures.md
index 6dae3a349b81..b7eb6e614352 100644
--- a/website/versioned_docs/version-1.0.0/procedures.md
+++ b/website/versioned_docs/version-1.0.0/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations.md#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations.md#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations.md#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations.md#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1535,11 +1535,11 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
-| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
-| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [clean_policy](configurations.md#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum qu [...]
+| [retain_commits](configurations.md#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations.md#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of [...]
+| [file_versions_retained](configurations.md#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations.md#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
@@ -1635,12 +1635,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations.md#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations.md#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations.md#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations.md#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations.md#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations.md#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1850,18 +1850,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations.md#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations.md#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations.md#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
+| [parallelism](configurations.md#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-1.0.0/python-rust-quick-start-guide.md b/website/versioned_docs/version-1.0.0/python-rust-quick-start-guide.md
index 3cabe6f1726e..fcd31e544200 100644
--- a/website/versioned_docs/version-1.0.0/python-rust-quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.0/python-rust-quick-start-guide.md
@@ -22,7 +22,7 @@ cargo add hudi
> [!NOTE]
> These examples expect a Hudi table exists at `/tmp/trips_table`, created using
-> the [quick start guide](quick-start-guide).
+> the [quick start guide](quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-1.0.0/querying_data.md b/website/versioned_docs/version-1.0.0/querying_data.md
index 66f836c956b2..f1e8fcc28dee 100644
--- a/website/versioned_docs/version-1.0.0/querying_data.md
+++ b/website/versioned_docs/version-1.0.0/querying_data.md
@@ -7,10 +7,10 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl.md), [SQL DML](sql_dml.md), [SQL Queries](sql_queries.md) and [Procedures](procedures.md) for the latest documentation.
:::
-Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts#query-types).
+Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts.md#query-types).
Once the table is synced to the Hive metastore, it provides external Hive tables backed by Hudi's custom inputformats. Once the proper hudi
bundle has been installed, the table can be queried by popular query engines like Hive, Spark SQL, Flink, Trino and PrestoDB.
@@ -19,7 +19,7 @@ In sections, below we will discuss specific setup to access different query type
## Spark Datasource
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
-See the [Spark Quick Start](quick-start-guide) for more examples of Spark datasource reading queries.
+See the [Spark Quick Start](quick-start-guide.md) for more examples of Spark datasource reading queries.
**Setup**
@@ -56,8 +56,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Incremental Queries](quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](configurations#SPARK_DATASOURCE) section, to view all datasource options.
+For examples, refer to [Incremental Queries](quick-start-guide.md#incremental-query) in the Spark quickstart.
+Please refer to [configurations](configurations.md#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-1.0.0/quick-start-guide.md b/website/versioned_docs/version-1.0.0/quick-start-guide.md
index 9a45e13c3f56..48dec4a0081e 100644
--- a/website/versioned_docs/version-1.0.0/quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.0/quick-start-guide.md
@@ -214,7 +214,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl.md) reference guide.
</TabItem>
@@ -258,7 +258,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -291,7 +291,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -300,7 +300,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml.md#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -412,7 +412,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml.md#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -442,7 +442,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts.md).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -545,7 +545,7 @@ Notice, instead of `UPDATE SET *`, we are updating only the `fare` column.
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data.md#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -625,7 +625,7 @@ Hudi supports indexing on columns to speed up queries. Indexes can be created on
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -1092,7 +1092,7 @@ Note that CDC queries are currently only supported on Copy-on-Write tables.
The examples thus far have showcased one of the two table types, that Hudi supports - Copy-on-Write (COW) tables. Hudi also supports
a more advanced write-optimized table type called Merge-on-Read (MOR) tables, that can balance read and write performance in a more
-flexible manner. See [table types](table_types) for more details.
+flexible manner. See [table types](table_types.md) for more details.
Any of these examples can be run on a Merge-on-Read table by simply changing the table type to MOR, while creating the table, as below.
@@ -1218,7 +1218,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations.md).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1230,8 +1230,8 @@ Hudi also uses this mechanism to support out-of-order data arrival into a table,
For e.g. using a _created_at_ timestamp field as the precombine field will prevent older versions of a record from overwriting newer ones or being exposed to queries, even
if they are written at a later commit time to the table. This is one of the key features, that makes Hudi, best suited for dealing with streaming data.
-To enable different merge semantics, Hudi supports [merge modes](record_merger). Commit time and event time based merge modes are supported out of the box.
-Users can also define their own custom merge strategies, see [here](sql_ddl#create-table-with-record-merge-mode).
+To enable different merge semantics, Hudi supports [merge modes](record_merger.md). Commit time and event time based merge modes are supported out of the box.
+Users can also define their own custom merge strategies, see [here](sql_ddl.md#create-table-with-record-merge-mode).
<Tabs
groupId="programming-language"
@@ -1294,29 +1294,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide.md).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
-For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl.md) and [Spark SQL DML](sql_dml.md) reference guides.
+For alter table commands, check out [this](sql_ddl.md#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures.md) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion.md#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes#spark-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes.md#spark-streaming) for more.
-Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data) and [streaming writes](writing_tables_streaming_writes).
+Check out more information on [modeling data in Hudi](faq_general.md#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data.md) and [streaming writes](writing_tables_streaming_writes.md).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it.
+steps [here](docker_demo.md) to get a taste for it.
diff --git a/website/versioned_docs/version-1.0.0/record_merger.md b/website/versioned_docs/version-1.0.0/record_merger.md
index f24478057be5..b88dbe1d00f2 100644
--- a/website/versioned_docs/version-1.0.0/record_merger.md
+++ b/website/versioned_docs/version-1.0.0/record_merger.md
@@ -6,7 +6,7 @@ toc_min_heading_level: 2
toc_max_heading_level: 4
---
-Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline#ordering-of-actions) section.
+Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline.md#ordering-of-actions) section.
To provide users full-fledged support for stream processing, Hudi goes all the way making the storage engine and the underlying storage format
understand how to merge changes to the same record key, that may arrive even in different order at different times. With the rise of mobile applications
and IoT, these scenarios have become the normal than an exception. For e.g. a social networking application uploading user events several hours after they happened,
@@ -54,7 +54,7 @@ With event time ordering, the merging picks the record with the highest value on
In the example above, two microservices product change records about orders at different times, that can arrive out-of-order. As color coded,
this can lead to application-level inconsistent states in the table if simply merged in commit time order like a cancelled order being re-created or
a paid order moved back to just created state expecting payment again. Event time ordering helps by ignoring older state changes that arrive late and
-avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control#non-blocking-concurrency-control-mode),
+avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control.md#non-blocking-concurrency-control-mode),
this provides a very powerful way for processing such data streams efficiently and correctly.
### CUSTOM
@@ -175,7 +175,7 @@ hoodie.datasource.write.payload.class=org.apache.hudi.common.model.EventTimeAvro
```
This is the default record payload for Flink based writing. Some use cases require merging records by event time and
thus event time plays the role of an ordering field. This payload is particularly useful in the case of late-arriving data.
-For such use cases, users need to set the [payload event time field](configurations#RECORD_PAYLOAD) configuration.
+For such use cases, users need to set the [payload event time field](configurations.md#RECORD_PAYLOAD) configuration.
#### OverwriteNonDefaultsWithLatestAvroPayload
```scala
@@ -249,5 +249,5 @@ Payload class can be specified using the below configs. For more advanced config
There are also quite a few other implementations. Developers may be interested in looking at the hierarchy of `HoodieRecordPayload` interface. For
example, [`MySqlDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/MySqlDebeziumAvroPayload.java) and [`PostgresDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/PostgresDebeziumAvroPayload.java) provides support for seamlessly applying changes
captured via Debezium for MySQL and PostgresDB. [`AWSDmsAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/AWSDmsAvroPayload.java) provides support for applying changes captured via Amazon Database Migration Service onto S3.
-For full configurations, go [here](configurations#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
+For full configurations, go [here](configurations.md#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
diff --git a/website/versioned_docs/version-1.0.0/s3_hoodie.md b/website/versioned_docs/version-1.0.0/s3_hoodie.md
index 3161ea4bd284..fadcb4b01c97 100644
--- a/website/versioned_docs/version-1.0.0/s3_hoodie.md
+++ b/website/versioned_docs/version-1.0.0/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](cleaning) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](cleaning.md) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.0/sql_ddl.md b/website/versioned_docs/version-1.0.0/sql_ddl.md
index d1a08846ecc0..565a62f480ef 100644
--- a/website/versioned_docs/version-1.0.0/sql_ddl.md
+++ b/website/versioned_docs/version-1.0.0/sql_ddl.md
@@ -75,7 +75,7 @@ should be specified as `PARTITIONED BY (dt, hh)`.
### Create table with record keys and ordering fields
-As discussed [here](quick-start-guide#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
Typically, this is also accompanied by configuring a `preCombineField` option to deal with out-of-order data and potential
duplicate records with the same key in the incoming writes.
@@ -105,7 +105,7 @@ TBLPROPERTIES (
### Create table with merge modes {#create-table-with-record-merge-mode}
-Hudi supports different [record merge modes](record_merger) to handle merge of incoming records with existing
+Hudi supports different [record merge modes](record_merger.md) to handle merge of incoming records with existing
records. To create a table with specific record merge mode, you can set `recordMergeMode` option.
```sql
@@ -127,7 +127,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode/';
With `EVENT_TIME_ORDERING`, the record with the larger event time (`precombineField`) overwrites the record with the
smaller event time on the same key, regardless of transaction's commit time. Users can set `CUSTOM` mode to provide their own
merge logic. With `CUSTOM` merge mode, you can provide a custom class that implements the merge logic. The interfaces
-to implement is explained in detail [here](record_merger#custom).
+to implement is explained in detail [here](record_merger.md#custom).
```sql
CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
@@ -147,7 +147,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion.md#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -236,7 +236,7 @@ AS SELECT * FROM parquet_table;
### Create Index
Hudi supports creating and dropping different types of indexes on a table. For more information on different
-type of indexes please refer [multi-modal indexing](indexes#multi-modal-indexing). Secondary
+type of indexes please refer [multi-modal indexing](indexes.md#multi-modal-indexing). Secondary
index, expression index and record indexes can be created using SQL create index command.
```sql
@@ -270,7 +270,7 @@ Both index and column on which the index is created can be qualified with some o
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
3. Secondary index is not supported for [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types).
:::
@@ -344,7 +344,7 @@ CREATE INDEX ts_hour ON hudi_table
:::note
1. Expression index can only be created for Spark engine using SQL. It is not supported yet with Spark DataSource API.
2. Expression index is not yet supported for [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types).
-3. Expression index is supported for unary and certain binary expressions. Please check [SQL DDL docs](sql_ddl#create-expression-index) for more details.
+3. Expression index is supported for unary and certain binary expressions. Please check [SQL DDL docs](sql_ddl.md#create-expression-index) for more details.
:::
The `expr` option is required for creating expression index, and it should be a valid Spark SQL function. Please check the syntax
@@ -577,7 +577,7 @@ Users can set table properties while creating a table. The important table prope
| Parameter Name | Default | Description |
|------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types) |
+| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types.md) |
| primaryKey | uuid | The primary key field names of the table separated by commas. Same as `hoodie.datasource.write.recordkey.field`. If this config is ignored, hudi will auto-generate primary keys. If explicitly set, primary key generation will honor user configuration. |
| preCombineField | | The pre-combine field of the table. It is used for resolving the final version of the record among multiple versions. Generally, `event time` or another similar column will be used for ordering purposes. Hudi will be able to handle out-of-order data using the preCombine field value. |
@@ -588,7 +588,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control.md#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -839,7 +839,7 @@ WITH (
### Create Table in Non-Blocking Concurrency Control Mode
-The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control#non-blocking-concurrency-control).
+The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control.md#non-blocking-concurrency-control).
```sql
-- This is a datagen source that can generate records continuously
@@ -907,7 +907,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations.md#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-1.0.0/sql_dml.md b/website/versioned_docs/version-1.0.0/sql_dml.md
index bf37711541b2..75c4544c023c 100644
--- a/website/versioned_docs/version-1.0.0/sql_dml.md
+++ b/website/versioned_docs/version-1.0.0/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
### Insert Into
@@ -29,7 +29,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -49,7 +49,7 @@ INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000
```
:::note Mapping to write operations
-Hudi offers flexibility in choosing the underlying [write operation](write_operations) of a `INSERT INTO` statement using
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
and *"upsert"* (with deduplication/merging). If a precombine field is not set, *"insert"* is chosen as the default. For a table with preCombine field set,
*"upsert"* is chosen as the default operation.
@@ -398,7 +398,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control.md#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-1.0.0/sql_queries.md b/website/versioned_docs/version-1.0.0/sql_queries.md
index 1ab1f1dc8289..aca0a0ac8ccc 100644
--- a/website/versioned_docs/version-1.0.0/sql_queries.md
+++ b/website/versioned_docs/version-1.0.0/sql_queries.md
@@ -8,11 +8,11 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-Hudi stores and organizes data on storage while providing different ways of [querying](concepts#query-types), across a wide range of query engines.
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
This page will show how to issue different queries and discuss any specific instructions for each query engine.
## Spark SQL
-The Spark [quickstart](quick-start-guide) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
@@ -90,7 +90,7 @@ show the files scanned as 1 compared to 3 files scanned without index.
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -196,7 +196,7 @@ DROP INDEX partition_stats on hudi_indexed_table;
### Snapshot Query with Event Time Ordering
-Hudi supports different [record merge modes](record_merger) for merging the records from the same key. Event
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
event time ordering merge mode.
@@ -323,7 +323,7 @@ For e.g the table has received 10 million modifications across 1 million records
see all changes in a given time window and not just the latest values.
:::
-Please refer to [configurations](basic_configurations) section for the important configuration options.
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
@@ -406,7 +406,7 @@ and the corresponding type column value.
## Flink SQL
Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
-relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide).
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-1.0.0/storage_layouts.md b/website/versioned_docs/version-1.0.0/storage_layouts.md
index 3b7911a15a1d..a88f584e3f9d 100644
--- a/website/versioned_docs/version-1.0.0/storage_layouts.md
+++ b/website/versioned_docs/version-1.0.0/storage_layouts.md
@@ -12,8 +12,8 @@ The following describes the general organization of files in storage for a Hudi
* Each slice contains a **_base file_** (parquet/orc/hfile) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) )
written by a commit that completed at a certain instant, along with set of **_log files_** (*.log.*) written by commits that completed
before the next base file's requested instant.
-* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
- file slices and [cleaning](cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+ file slices and [cleaning](cleaning.md) action gets rid of unused/older file slices to reclaim space on the file system.
* All metadata including timeline, metadata table are stored in a special `.hoodie` directory under the base path.
diff --git a/website/versioned_docs/version-1.0.0/structure.md b/website/versioned_docs/version-1.0.0/structure.md
index 31281b68f71c..9f78f89f16f5 100644
--- a/website/versioned_docs/version-1.0.0/structure.md
+++ b/website/versioned_docs/version-1.0.0/structure.md
@@ -17,4 +17,4 @@ Hudi (pronounced βHoodieβ) ingests & manages storage of large analytical tab
By carefully managing how data is laid out in storage & how itβs exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [PrestoDB](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) [...]
-Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide) for a demo.
+Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide.md) for a demo.
diff --git a/website/versioned_docs/version-1.0.0/syncing_aws_glue_data_catalog.md b/website/versioned_docs/version-1.0.0/syncing_aws_glue_data_catalog.md
index b6f6c82a6c54..35f43a8af472 100644
--- a/website/versioned_docs/version-1.0.0/syncing_aws_glue_data_catalog.md
+++ b/website/versioned_docs/version-1.0.0/syncing_aws_glue_data_catalog.md
@@ -10,7 +10,7 @@ and send them to AWS Glue.
## Configurations
Most of the configurations for `AwsGlueCatalogSyncTool` are shared with `HiveSyncTool`. The example showed in
-[Sync to Hive Metastore](syncing_metastore) can be used as is for sync with Glue Data Catalog, provided that the hive metastore
+[Sync to Hive Metastore](syncing_metastore.md) can be used as is for sync with Glue Data Catalog, provided that the hive metastore
URL (either JDBC or thrift URI) can proxied to Glue Data Catalog, which is usually done within AWS EMR or Glue job environment.
For Hudi streamer, users can set
diff --git a/website/versioned_docs/version-1.0.0/syncing_metastore.md b/website/versioned_docs/version-1.0.0/syncing_metastore.md
index d29eea66f8b0..f7c15692f251 100644
--- a/website/versioned_docs/version-1.0.0/syncing_metastore.md
+++ b/website/versioned_docs/version-1.0.0/syncing_metastore.md
@@ -124,7 +124,7 @@ will be inferred and set automatically.
## Hive Sync Tool
-Writing data with [DataSource](writing_data) writer or [HoodieStreamer](hoodie_streaming_ingestion) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
+Writing data with [DataSource](writing_data.md) writer or [HoodieStreamer](hoodie_streaming_ingestion.md) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
In case, it's preferable to run this from commandline or in an independent jvm, Hudi provides a `HiveSyncTool`, which can be invoked as below,
once you have built the hudi-hive module. Following is how we sync the above Datasource Writer written table to Hive metastore.
diff --git a/website/versioned_docs/version-1.0.0/use_cases.md b/website/versioned_docs/version-1.0.0/use_cases.md
index 2194e3108e60..251df6187611 100644
--- a/website/versioned_docs/version-1.0.0/use_cases.md
+++ b/website/versioned_docs/version-1.0.0/use_cases.md
@@ -19,7 +19,7 @@ it is unfortunately often solved in a piecemeal fashion, using a medley of inges
### Why Hudi?
- Unique design choices like Merge-On-Read tables, record-level indexes and asynchronous compaction, approach theoretical optimality for absorbing changes to tables quickly and efficiently.
-- Built-in ingestion tools on [Spark](hoodie_streaming_ingestion), [Flink](ingestion_flink) and [Kafka Connect](ingestion_kafka_connect), that let you ingest data with a single command.
+- Built-in ingestion tools on [Spark](hoodie_streaming_ingestion.md), [Flink](ingestion_flink.md) and [Kafka Connect](ingestion_kafka_connect.md), that let you ingest data with a single command.
- Support for incremental ingestion with automatic checkpoint management from streaming sources (Kafka, Pulsar, ...), Cloud storage (S3, GCS, ADLS, etc.) and even JDBC.
- Support for widely used data formats (Protobuf, Avro, JSON), file formats (parquet, orc, avro, etc.) and change log formats like [Debezium](http://debezium.io/).
- Even for scalable de-duplication for high-volume append-only streaming data, by employing bloom filter indexes and advanced data structures like interval trees for efficient range pruning.
@@ -65,10 +65,10 @@ together to build out the platform. Such an open platform is also essential for
- While open data formats help, Hudi unlocks complete freedom by also providing open compute services for ingesting, optimizing, indexing and querying data. For e.g Hudi's writers come with
a self-managing table service runtime that can maintain tables automatically in the background on each write. Often times, Hudi and your favorite open query engine is all
you need to get an open data platform up and running.
-- Examples of open services that make performance optimization or management easy include: [auto file sizing](file_sizing) to solve the "small files" problem,
- [clustering](clustering) to co-locate data next to each other, [compaction](compaction) to allow tuning of low latency ingestion + fast read queries,
- [indexing](indexes) - for faster writes/queries, Multi-Dimensional Partitioning (Z-Ordering), automatic cleanup of uncommitted data with marker mechanism,
- [auto cleaning](cleaning) to automatically removing old versions of files.
+- Examples of open services that make performance optimization or management easy include: [auto file sizing](file_sizing.md) to solve the "small files" problem,
+ [clustering](clustering.md) to co-locate data next to each other, [compaction](compaction.md) to allow tuning of low latency ingestion + fast read queries,
+ [indexing](indexes.md) - for faster writes/queries, Multi-Dimensional Partitioning (Z-Ordering), automatic cleanup of uncommitted data with marker mechanism,
+ [auto cleaning](cleaning.md) to automatically removing old versions of files.
- Hudi provides rich options for pre-sorting/loading data efficiently and then follow on with rich set of data clustering techniques to manage file sizes and data distribution within a table. In each case, Hudi provides high-degree of configurability in terms of when/how often these services are scheduled, planned and executed. For e.g. Hudi ships with a handful of common planning strategies for compaction and clustering.
- Along with compatibility with other open table formats like [Apache Iceberg](https://iceberg.apache.org/)/[Delta Lake](https://delta.io/), and catalog sync services to various data catalogs, Hudi is one of the most open choices for your data foundation.
diff --git a/website/versioned_docs/version-1.0.0/write_operations.md b/website/versioned_docs/version-1.0.0/write_operations.md
index 141dc531de50..b8d66a0bf393 100644
--- a/website/versioned_docs/version-1.0.0/write_operations.md
+++ b/website/versioned_docs/version-1.0.0/write_operations.md
@@ -119,17 +119,17 @@ Here are the basic configs relevant to the write operations types mentioned abov
## Write path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](configurations#hoodiecombinebeforeinsert) : First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](indexes) : Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](file_sizing): Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](storage_layouts): We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
+1. [Deduping](configurations.md#hoodiecombinebeforeinsert) : First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
+2. [Index Lookup](indexes.md) : Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
+3. [File Sizing](file_sizing.md): Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
+4. [Partitioning](storage_layouts.md): We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O :Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](indexes): Now that the write is performed, we will go back and update the index.
-7. Commit: Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback) can be configured.)
-8. [Clean](cleaning) (if needed): Following the commit, cleaning is invoked if needed.
-9. [Compaction](compaction): If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
-10. Archive : Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
+6. Update [Index](indexes.md): Now that the write is performed, we will go back and update the index.
+7. Commit: Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback.md) can be configured.)
+8. [Clean](cleaning.md) (if needed): Following the commit, cleaning is invoked if needed.
+9. [Compaction](compaction.md): If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
+10. Archive : Lastly, we perform an archival step which moves old [timeline](timeline.md) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-1.0.0/writing_data.md b/website/versioned_docs/version-1.0.0/writing_data.md
index edf26430a06f..965903d04f14 100644
--- a/website/versioned_docs/version-1.0.0/writing_data.md
+++ b/website/versioned_docs/version-1.0.0/writing_data.md
@@ -28,15 +28,15 @@ Default value: `"partitionpath"`<br/>
**PRECOMBINE_FIELD**: When two records within the same batch have the same key value, the record with the largest value from the field specified will be choosen. If you are using default payload of OverwriteWithLatestAvroPayload for HoodieRecordPayload (`WRITE_PAYLOAD_CLASS`), an incoming record will always takes precendence compared to the one in storage ignoring this `PRECOMBINE_FIELD_OPT_KEY`. <br/>
Default value: `"ts"`<br/>
-**OPERATION**: The [write operations](write_operations) to use.<br/>
+**OPERATION**: The [write operations](write_operations.md) to use.<br/>
Available values:<br/>
`"upsert"` (default), `"bulk_insert"`, `"insert"`, `"delete"`
-**TABLE_TYPE**: The [type of table](concepts#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
+**TABLE_TYPE**: The [type of table](concepts.md#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
Available values:<br/>
-[`COW_TABLE_TYPE_OPT_VAL`](concepts#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts#merge-on-read-table)
+[`COW_TABLE_TYPE_OPT_VAL`](concepts.md#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts.md#merge-on-read-table)
-**KEYGENERATOR_CLASS_NAME**: Refer to [Key Generation](key_generation) section below.
+**KEYGENERATOR_CLASS_NAME**: Refer to [Key Generation](key_generation.md) section below.
Example:
@@ -84,9 +84,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -120,9 +120,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -421,7 +421,7 @@ Then any record you want to delete you can mark `_hoodie_is_deleted` as true:
Following is an example of how to use `optimistic_concurrency_control` via Spark DataSource API.
-Read more in-depth details about concurrency control in the [concurrency control concepts](concurrency_control) section.
+Read more in-depth details about concurrency control in the [concurrency control concepts](concurrency_control.md) section.
```java
inputDF.write.format("hudi")
diff --git a/website/versioned_docs/version-1.0.1/basic_configurations.md b/website/versioned_docs/version-1.0.1/basic_configurations.md
index 65f404721017..c823c2522f44 100644
--- a/website/versioned_docs/version-1.0.1/basic_configurations.md
+++ b/website/versioned_docs/version-1.0.1/basic_configurations.md
@@ -1,11 +1,11 @@
---
title: Basic Configurations
-summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
last_modified_at: 2024-12-06T17:38:05.854
---
-This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
diff --git a/website/versioned_docs/version-1.0.1/cleaning.md b/website/versioned_docs/version-1.0.1/cleaning.md
index e98ce63f6055..203931bb3605 100644
--- a/website/versioned_docs/version-1.0.1/cleaning.md
+++ b/website/versioned_docs/version-1.0.1/cleaning.md
@@ -133,7 +133,7 @@ spark-submit --master local \
Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
#### CLI
-You can also use [Hudi CLI](cli) to run Hoodie Cleaner.
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
CLI provides the below commands for cleaner service:
- `cleans show`
diff --git a/website/versioned_docs/version-1.0.1/cli.md b/website/versioned_docs/version-1.0.1/cli.md
index 54b7753626b8..1fc1795c777a 100644
--- a/website/versioned_docs/version-1.0.1/cli.md
+++ b/website/versioned_docs/version-1.0.1/cli.md
@@ -577,7 +577,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-1.0.1/cloud.md b/website/versioned_docs/version-1.0.1/cloud.md
index 123abd5e6bea..a3477b551bee 100644
--- a/website/versioned_docs/version-1.0.1/cloud.md
+++ b/website/versioned_docs/version-1.0.1/cloud.md
@@ -11,23 +11,23 @@ last_modified_at: 2021-10-12T10:50:00+08:00
Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
to cloud stores.
-* [AWS S3](s3_hoodie) <br/>
+* [AWS S3](s3_hoodie.md) <br/>
Configurations required for S3 and Hudi co-operability.
-* [Google Cloud Storage](gcs_hoodie) <br/>
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
Configurations required for GCS and Hudi co-operability.
-* [Alibaba Cloud OSS](oss_hoodie) <br/>
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
Configurations required for OSS and Hudi co-operability.
-* [Microsoft Azure](azure_hoodie) <br/>
+* [Microsoft Azure](azure_hoodie.md) <br/>
Configurations required for Azure and Hudi co-operability.
-* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
Configurations required for COS and Hudi co-operability.
-* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
Configurations required for IBM Cloud Object Storage and Hudi co-operability.
-* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
Configurations required for BOS and Hudi co-operability.
-* [JuiceFS](jfs_hoodie) <br/>
+* [JuiceFS](jfs_hoodie.md) <br/>
Configurations required for JuiceFS and Hudi co-operability.
-* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
Configurations required for OCI and Hudi co-operability.
:::note
diff --git a/website/versioned_docs/version-1.0.1/clustering.md b/website/versioned_docs/version-1.0.1/clustering.md
index dea8d507009a..24d7333a7739 100644
--- a/website/versioned_docs/version-1.0.1/clustering.md
+++ b/website/versioned_docs/version-1.0.1/clustering.md
@@ -13,7 +13,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## How is compaction different from clustering?
Hudi is modeled like a log-structured storage engine with multiple versions of the data.
-Particularly, [Merge-On-Read](table_types#merge-on-read-table)
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
@@ -93,7 +93,7 @@ update strategy.
This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
-config [hoodie.clustering.plan.strategy.class](configurations#hoodieclusteringplanstrategyclass).
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
Different plan strategies are as follows:
diff --git a/website/versioned_docs/version-1.0.1/compaction.md b/website/versioned_docs/version-1.0.1/compaction.md
index b72978239688..f984c05af1e1 100644
--- a/website/versioned_docs/version-1.0.1/compaction.md
+++ b/website/versioned_docs/version-1.0.1/compaction.md
@@ -13,7 +13,7 @@ not applicable to Copy On Write(COW) tables and only applies to MOR tables.
### Why MOR tables need compaction?
To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
-data is organized in terms of [file groups](storage_layouts). Each file group in a MOR table
+data is organized in terms of [file groups](storage_layouts.md). Each file group in a MOR table
consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
are appended to log files.
@@ -182,7 +182,7 @@ This model may need a lock provider **if** metadata table is enabled.
:::
#### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
The compactor utility allows to do scheduling and execution of compaction.
Example:
@@ -199,7 +199,7 @@ Note, the `instant-time` parameter is now optional for the Hudi Compactor Utilit
the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
#### Hudi CLI
-Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-1.0.1/concurrency_control.md b/website/versioned_docs/version-1.0.1/concurrency_control.md
index d835e06290f9..b84a3752815e 100644
--- a/website/versioned_docs/version-1.0.1/concurrency_control.md
+++ b/website/versioned_docs/version-1.0.1/concurrency_control.md
@@ -8,7 +8,7 @@ last_modified_at: 2021-03-19T15:59:57-04:00
---
Concurrency control defines how different writers/readers/table services coordinate access to a Hudi table. Hudi ensures atomic writes, by way of publishing commits atomically to the timeline,
stamped with an instant time that denotes the time at which the action is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between
-writer processes that issue [write operations](write_operations) and table services that (re)write data/metadata to optimize/perform bookkeeping and
+writer processes that issue [write operations](write_operations.md) and table services that (re)write data/metadata to optimize/perform bookkeeping and
readers (that execute queries and read data).
Hudi provides
@@ -227,7 +227,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.1-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](sql_dml#non-blocking-concurrency-control-experimental).
+writers [here](sql_dml.md#non-blocking-concurrency-control-experimental).
:::note
`NON_BLOCKING_CONCURRENCY_CONTROL` between ingestion writer and table service writer is not yet supported for clustering.
diff --git a/website/versioned_docs/version-1.0.1/deployment.md b/website/versioned_docs/version-1.0.1/deployment.md
index 1e0df56da363..f617f20cac64 100644
--- a/website/versioned_docs/version-1.0.1/deployment.md
+++ b/website/versioned_docs/version-1.0.1/deployment.md
@@ -26,13 +26,13 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### Hudi Streamer
-[Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
To use Hudi Streamer in Spark, the `hudi-utilities-slim-bundle` and Hudi Spark bundle are required, by adding
`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.1,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.1` to the `spark-submit` command.
- - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -133,7 +133,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Batch Writes](writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.d [...]
+As described in [Batch Writes](writing_data.md#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
@@ -260,7 +260,7 @@ hoodie.table.version=5
```
Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
-For more details on downgrade, please refer documentation [here](cli#upgrade-and-downgrade-table).
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
```shell
connect --path /tmp/hudi_table
downgrade table --toVersion 4
@@ -281,4 +281,4 @@ Currently migrating to Hudi can be done using two approaches
- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
-For more details, refer to the detailed [migration guide](migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-1.0.1/faq.md b/website/versioned_docs/version-1.0.1/faq.md
index 26c3eb50d214..c53c8ebbc68c 100644
--- a/website/versioned_docs/version-1.0.1/faq.md
+++ b/website/versioned_docs/version-1.0.1/faq.md
@@ -6,10 +6,10 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](faq_general)
-- [Design & Concepts](faq_design_and_concepts)
-- [Writing Tables](faq_writing_tables)
-- [Reading Tables](faq_reading_tables)
-- [Table Services](faq_table_services)
-- [Storage](faq_storage)
-- [Integrations](faq_integrations)
+- [General](faq_general.md)
+- [Design & Concepts](faq_design_and_concepts.md)
+- [Writing Tables](faq_writing_tables.md)
+- [Reading Tables](faq_reading_tables.md)
+- [Table Services](faq_table_services.md)
+- [Storage](faq_storage.md)
+- [Integrations](faq_integrations.md)
diff --git a/website/versioned_docs/version-1.0.1/faq_design_and_concepts.md b/website/versioned_docs/version-1.0.1/faq_design_and_concepts.md
index 720f4ae8e2df..8aee10defc1a 100644
--- a/website/versioned_docs/version-1.0.1/faq_design_and_concepts.md
+++ b/website/versioned_docs/version-1.0.1/faq_design_and_concepts.md
@@ -6,14 +6,14 @@ keywords: [hudi, writing, reading]
### How does Hudi ensure atomicity?
-Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
+Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline.md) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
### Does Hudi extend the Hive table layout?
Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
-* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
-* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users exper [...]
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
+* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering.md) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users ex [...]
* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudiβs design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
### What concurrency control approaches does Hudi adopt?
@@ -32,7 +32,7 @@ In these scenarios, it might be tempting to think of data inconsistencies/data l
(examples [1](https://github.com/apache/hudi/blob/aea5bb6f0ab824247f5e3498762ad94f643a2cb6/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L76),
[2](https://github.com/apache/hudi/blame/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L173)) in incremental queries to ensure that no data
is served beyond the point there is an inflight instant in its timeline, so no data loss or drop happens. This detection is made possible because Hudi writes first request a transaction on the timeline, before planning/executing
-the write, as explained in the [timeline](timeline#states) section.
+the write, as explained in the [timeline](timeline.md#states) section.
In this case, on seeing C1βs inflight commit (publish to timeline is atomic), C2 data (which is > C1 in the timeline) is not served until C1 inflight transitions to a terminal state such as completed or marked as failed.
This [test](https://github.com/apache/hudi/blob/master/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java#L137) demonstrates how Hudi incremental source stops proceeding until C1 completes.
@@ -48,7 +48,7 @@ To expand more on the long term approach, Hudi has had a proposal to streamline/
This has been delayed for a few reasons
- Large hosted query engines and users not upgrading fast enough.
-- The issues brought up - \[[1](faq_design_and_concepts#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
+- The issues brought up - \[[1](faq_design_and_concepts.md#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts.md#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
relevant to this are not practically very important to users beyond good pedantic discussions,
- Wanting to do it alongside [non-blocking concurrency control](https://github.com/apache/hudi/pull/7907) in Hudi version 1.x.
diff --git a/website/versioned_docs/version-1.0.1/faq_general.md b/website/versioned_docs/version-1.0.1/faq_general.md
index cb9e0161bcb8..bfefd3b6d851 100644
--- a/website/versioned_docs/version-1.0.1/faq_general.md
+++ b/website/versioned_docs/version-1.0.1/faq_general.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### When is Hudi useful for me or my organization?
-If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
+If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion.md). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
@@ -37,7 +37,7 @@ Choose Copy-on-write storage if :
* You are looking for a simple alternative, that replaces your existing parquet tables without any need for real-time data.
* Your current job is rewriting entire table/partition to deal with updates, while only a few files actually change in each partition.
-* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
+* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations.md#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
* Your workload is fairly well-understood and does not have sudden bursts of large amount of update or inserts to older partitions. COW absorbs all the merging cost on the writer side and thus these sudden changes can clog up your ingestion and interfere with meeting normal mode ingest latency targets.
Choose merge-on-read storage if :
@@ -51,7 +51,7 @@ Immaterial of what you choose, Hudi provides
* Incremental pulls
* Ability to de-duplicate data
-Find more [here](concepts/).
+Find more [here](concepts.md).
### Is Hudi an analytical database?
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data.md) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
@@ -79,7 +79,7 @@ generate the key internally (much like RDBMS row_ids)
### How does Hudi actually store data inside a table?
-At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts/) for more details.
+At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts.md) for more details.
### How Hudi handles partition evolution requirements ?
diff --git a/website/versioned_docs/version-1.0.1/faq_reading_tables.md b/website/versioned_docs/version-1.0.1/faq_reading_tables.md
index 3f413b7ef1e0..e9db97dbbdd2 100644
--- a/website/versioned_docs/version-1.0.1/faq_reading_tables.md
+++ b/website/versioned_docs/version-1.0.1/faq_reading_tables.md
@@ -24,7 +24,7 @@ set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
### Does Hudi guarantee consistent reads? How to think about read optimized queries?
-Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
+Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types.md#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
Hudiβs read-optimized query is targeted for the MOR table only, with guidance around how compaction should be run to achieve predictable results. In the MOR table, the compaction, which runs every few commits (or βdeltacommitβ to be exact for the MOR table) by default, merges the base (parquet) file and corresponding change log files to a new base file within each file group, so that the snapshot query serving the latest data immediately after compaction reads the base files only.Β Simil [...]
diff --git a/website/versioned_docs/version-1.0.1/faq_storage.md b/website/versioned_docs/version-1.0.1/faq_storage.md
index b088b30d8ebd..2f6950fed334 100644
--- a/website/versioned_docs/version-1.0.1/faq_storage.md
+++ b/website/versioned_docs/version-1.0.1/faq_storage.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### Does Hudi support cloud storage/object stores?
-Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
+Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud.md) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations.md#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
### What is the difference between copy-on-write (COW) vs merge-on-read (MOR) table types?
@@ -14,11 +14,11 @@ Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop
**Merge On Read** - This storage type enables clients to ingest data quickly onto row based data format such as avro. Any new data that is written to the Hudi table using MOR table type, will write new log/delta files that internally store the data as avro encoded bytes. A compaction process (configured as inline or asynchronous) will convert log file format to columnar file format (parquet). Two different InputFormats expose 2 different views of this data, Read Optimized view exposes co [...]
-More details can be found [here](concepts/) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
+More details can be found [here](concepts.md) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
### How do I migrate my data to Hudi?
-Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide/), including ways to doing partial migrations.
+Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables.md#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide.md), including ways to doing partial migrations.
### How to convert an existing COW table to MOR?
@@ -34,7 +34,7 @@ But manually changing it will result in checksum errors. So, we have to go via h
### How can I find the average record size in a commit?
-The `commit showpartitons` command in [HUDI CLI](cli) will show both "bytes written" and
+The `commit showpartitons` command in [HUDI CLI](cli.md) will show both "bytes written" and
"records inserted." Divide the bytes written by records inserted to find the average size. Note that this answer assumes
@@ -91,11 +91,11 @@ spark.read.parquet("your_data_set/path/to/month").limit(n) // Limit n records
.save(basePath);
```
-For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli).
+For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli.md).
### Why does maintain record level commit metadata? Isn't tracking table version at file level good enough?Β
-By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hudi [...]
+By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering.md#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hu [...]
### Why partition fields are also stored in parquet files in addition to the partition path ?
diff --git a/website/versioned_docs/version-1.0.1/faq_table_services.md b/website/versioned_docs/version-1.0.1/faq_table_services.md
index 7af1ec75398b..dc4dbcf9716f 100644
--- a/website/versioned_docs/version-1.0.1/faq_table_services.md
+++ b/website/versioned_docs/version-1.0.1/faq_table_services.md
@@ -6,11 +6,11 @@ keywords: [hudi, writing, reading]
### What does the Hudi cleaner do?
-The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be read [...]
+The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations.md#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be r [...]
### How do I run compaction for a MOR table?
-Simplest way to run compaction on MOR table is to run the [compaction inline](configurations#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted most o [...]
+Simplest way to run compaction on MOR table is to run the [compaction inline](configurations.md#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted mos [...]
That said, for obvious reasons of not blocking ingesting for compaction, you may want to run it asynchronously as well. This can be done either via a separate [compaction job](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java) that is scheduled by your workflow scheduler/notebook independently. If you are using Hudi Streamer, then you can run in [continuous mode](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca [...]
@@ -40,7 +40,7 @@ Depending on how you write to Hudi these are the possible options currently.
### How to disable all table services in case of multiple writers?
-[hoodie.table.services.enabled](configurations#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
+[hoodie.table.services.enabled](configurations.md#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
### Why does Hudi retain at-least one previous commit even after setting hoodie.cleaner.commits.retained': 1 ?
@@ -50,6 +50,6 @@ Hudi runs cleaner to remove old file versions as part of writing data either in
Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
-be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback)
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback.md)
for details
diff --git a/website/versioned_docs/version-1.0.1/faq_writing_tables.md b/website/versioned_docs/version-1.0.1/faq_writing_tables.md
index fe061d1b491a..94badc5daa81 100644
--- a/website/versioned_docs/version-1.0.1/faq_writing_tables.md
+++ b/website/versioned_docs/version-1.0.1/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API and us [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations.md) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API a [...]
### How is a Hudi writer job deployed?
@@ -68,21 +68,21 @@ As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github
### How do I delete records in the dataset using Hudi?
-GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data#deletes).
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data.md#deletes).
### Should I need to worry about deleting all copies of the records in case of duplicates?
-No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https://githu [...]
+No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables.md#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables.md#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https:/ [...]
### How does Hudi handle duplicate record keys in an input?
-When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
+When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
-For an insert or bulk_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in [...]
+For an insert or bulk_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or [...]
### How can I pass hudi configurations to my spark writer job?
-Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations/). Invoking _--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
+Hudi configuration options covering the datasource and low level Hudi write client (which both deltastreamer & datasource internally call) are [here](configurations.md). Invoking _--help_ on any tool such as DeltaStreamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
* For Spark DataSource, you can use the "options" API of DataFrameWriter to pass in these configs.
@@ -108,28 +108,28 @@ hoodie.datasource.write.hive_style_partitioning: true
### Can I register my Hudi table with Apache Hive metastore?
-Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations#hoodiedatasourcehive_syncenable).
+Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore.md#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations.md#hoodiedatasourcehive_syncenable).
### What's Hudi's schema evolution story?
Hudi uses Avro as the internal canonical representation for records, primarily due to its nice [schema compatibility & evolution](https://docs.confluent.io/platform/current/schema-registry/avro.html) properties. This is a key aspect of having reliability in your ingestion or ETL pipelines. As long as the schema passed to Hudi (either explicitly in Hudi Streamer schema provider configs or implicitly by Spark Datasource's Dataset schemas) is backwards compatible (e.g no field deletes, only [...]
-Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution) for more details on Schema-on-read for Spark..
+Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution.md) for more details on Schema-on-read for Spark..
### What performance/ingest latency can I expect for Hudi writing?
-The speed at which you can write into Hudi depends on the [write operation](write_operations) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when try [...]
+The speed at which you can write into Hudi depends on the [write operation](write_operations.md) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when [...]
| Storage Type | Type of workload | Performance | Tips |
| ---| ---| ---| --- |
-| copy on write | bulk_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism) to get right number of files. Use insert if you want this auto tuned. Configure [hoodie.bulkinsert.sort.mode](configurations#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value `NONE` offers the fastest performance and matches `spark.write.parquet()` [...]
-| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
-| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or less than 50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For example, if a partitio [...]
+| copy on write | bulk_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism) to get right number of files. Use insert if you want this auto tuned. Configure [hoodie.bulkinsert.sort.mode](configurations.md#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value `NONE` offers the fastest performance and matches `spark.write.parqu [...]
+| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations.md#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
+| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or less than 50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For example, if a partitio [...]
| merge on read | bulk insert | Currently new data only goes to parquet files and thus performance here should be similar to copy on write bulk insert. This has the nice side-effect of getting data into parquet directly for query performance. [This GitHub issue](https://github.com/apache/hudi/issues/14468) will add support for logging inserts directly and this up drastically. | |
| merge on read | insert | Similar to above | |
| merge on read | upsert/ de-duplicate & insert | Indexing performance would remain the same as copy-on-write, while ingest latency for updates (costliest I/O operation in copy on write) are sent to log files and thus with asynchronous compaction provides very good ingest performance with low write amplification. | |
-Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide) for more tips on JVM and other configurations.
+Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide.md) for more tips on JVM and other configurations.
### What performance can I expect for Hudi reading/queries?
@@ -147,11 +147,11 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations#hoodieparquetmaxfilesize) and the [soft limit](configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on previou [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations.md#hoodieparquetmaxfilesize) and the [soft limit](configurations.md#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on p [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
-* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations#hoodielogfilemaxsize) and a [factor](configurations#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
+* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations.md#hoodielogfilemaxsize) and a [factor](configurations.md#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
* Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations as above for the COPY_ON_WRITE case applies.
NOTE : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the compaction has been p [...]
@@ -173,7 +173,7 @@ hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPa
With each commit, Hudi creates a new table version in the metastore. This can be reduced by setting the option
-[hoodie.datasource.meta_sync.condition.sync](configurations#hoodiedatasourcemeta_syncconditionsync) to true.
+[hoodie.datasource.meta_sync.condition.sync](configurations.md#hoodiedatasourcemeta_syncconditionsync) to true.
This will ensure that hive sync is triggered on schema or partitions changes.
@@ -183,11 +183,11 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
+Hudi employs [optimistic concurrency control](concurrency_control.md) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/ [...]
### Can single-writer inserts have duplicates?
-By default, Hudi turns off key based de-duplication for INSERT/BULK_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in delta [...]
+By default, Hudi turns off key based de-duplication for INSERT/BULK_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in de [...]
### Can concurrent inserts cause duplicates?
diff --git a/website/versioned_docs/version-1.0.1/file_sizing.md b/website/versioned_docs/version-1.0.1/file_sizing.md
index 62ad0f7a4320..02ec64a8fe43 100644
--- a/website/versioned_docs/version-1.0.1/file_sizing.md
+++ b/website/versioned_docs/version-1.0.1/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning.md)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-1.0.1/flink-quick-start-guide.md b/website/versioned_docs/version-1.0.1/flink-quick-start-guide.md
index 6c592fa537b2..d0071cc57190 100644
--- a/website/versioned_docs/version-1.0.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.1/flink-quick-start-guide.md
@@ -60,7 +60,7 @@ quick start tool for SQL users.
### Start Flink SQL client
Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
-You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore#install)), or download it from the
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
Now start the SQL CLI:
@@ -282,7 +282,7 @@ Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-e
>
This statement queries snapshot view of the dataset.
-Refers to [Table types and queries](concepts#table-types--queries) for more info on all table types and query types supported.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
## Update Data
@@ -325,7 +325,7 @@ Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/ma
</Tabs
>
-[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts)
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
denoted by the timestamp.
@@ -449,19 +449,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink#cdc-ingestion), [Bulk Insert](ingestion_flink#bulk-insert), [Index Bootstrap](ingestion_flink#index-bootstrap), [Changelog Mode](ingestion_flink#changelog-mode) and [Append Mode](ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml#non-blocking-concurrency-control-experimental).
-- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](sql_queries#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
-- **Optimization**: Offline compaction is supported [Offline Compaction](compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk Insert](ingestion_flink.md#bulk-insert), [Index Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog Mode](ingestion_flink.md#changelog-mode) and [Append Mode](ingestion_flink.md#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml.md#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](sql_queries.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
- - [Hudi Storage Layout](storage_layouts) - How the files are laid out on storage
- - [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+ - [Hudi Storage Layout](storage_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -470,5 +470,5 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it. Also, if you are looking for ways to migrate your existing data
-to Hudi, refer to [migration guide](migration_guide).
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
index d12f294c92d4..5f559242e8b9 100644
--- a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
@@ -268,7 +268,7 @@ and then ingest it as follows.
--op BULK_INSERT
```
-In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide).
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
### Using `hudi-utilities-slim-bundle` bundle jar
@@ -303,7 +303,7 @@ A Hudi Streamer job can then be triggered as follows:
--op BULK_INSERT
```
-Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control) section
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
### Checkpointing
@@ -314,7 +314,7 @@ Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single [...]
+- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a sing [...]
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
diff --git a/website/versioned_docs/version-1.0.1/hudi_stack.md b/website/versioned_docs/version-1.0.1/hudi_stack.md
index 7aba8d5842fd..b8b063b17fdd 100644
--- a/website/versioned_docs/version-1.0.1/hudi_stack.md
+++ b/website/versioned_docs/version-1.0.1/hudi_stack.md
@@ -49,19 +49,19 @@ bring any compute engine for specific workloads.
Drawing an analogy to file formats, a table format simply concerns with how files are distributed with the table, partitioning schemes, schema and metadata tracking changes. Hudi organizes files within a table or partition into
File Groups. Updates are captured in log files tied to these File Groups, ensuring efficient merges. There are three major components related to Hudiβs table format.
-- **Timeline** : Hudi's [timeline](timeline), stored in the `/.hoodie/timeline` folder, is a crucial event log recording all table actions in an ordered manner,
+- **Timeline** : Hudi's [timeline](timeline.md), stored in the `/.hoodie/timeline` folder, is a crucial event log recording all table actions in an ordered manner,
with events kept for a specified period. Hudi uniquely designs each File Group as a self-contained log, enabling record state reconstruction through delta logs, even after archival of historical actions. This approach effectively limits metadata size based on table activity frequency, essential for managing tables with frequent updates.
- **File Group and File Slice** : Within each partition the data is physically stored as base and Log Files and organized into logical concepts as [File groups](https://hudi.apache.org/tech-specs-1point0/#storage-layout) and
File Slices. File groups contain multiple versions of File Slices and are split into multiple File Slices. A File Slice comprises the Base and Log File. Each File Slice within
the file-group is uniquely identified by the write that created its base file or the first log file, which helps order the File Slices.
-- **Metadata Table** : Implemented as another merge-on-read Hudi table, the [metadata table](metadata) efficiently handles quick updates with low write amplification.
+- **Metadata Table** : Implemented as another merge-on-read Hudi table, the [metadata table](metadata.md) efficiently handles quick updates with low write amplification.
It leverages a [SSTable](https://cassandra.apache.org/doc/stable/cassandra/architecture/storage-engine.html#sstables) based file format for quick, indexed key lookups,
storing vital information like file paths, column statistics and schema. This approach streamlines operations by reducing the necessity for expensive cloud file listings.
Hudiβs approach of recording updates into Log Files is more efficient and involves low merge overhead than systems like Hive ACID, where merging all delta records against
-all Base Files is required. Read more about the various table types in Hudi [here](table_types).
+all Base Files is required. Read more about the various table types in Hudi [here](table_types.md).
## Storage Engine
@@ -74,8 +74,8 @@ Cassandra and Clickhouse.
<p align = "center">Figure: Indexes in Hudi</p>
-[Indexes](indexes) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. The [metadata table](metadata/#metadata-table-indices) acts
-as an additional [indexing system](metadata#supporting-multi-modal-index-in-hudi) and brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata
+[Indexes](indexes.md) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. The [metadata table](metadata/#metadata-table-indices) acts
+as an additional [indexing system](metadata.md#supporting-multi-modal-index-in-hudi) and brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata
table, like file listings, column statistics, bloom filters, record-level indexes, and [expression indexes](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) to quickly generate optimized query plans and improve read
performance. In addition to the metadata table indexes, Hudi supports simple join based indexing, bloom filters stored in base file footers, external key-value stores like HBase,
and optimized storage techniques like bucketing , to efficiently locate File Groups containing specific record keys. Hudi also provides reader indexes such as [expression](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) and
@@ -91,12 +91,12 @@ running them in inline, semi-asynchronous or full-asynchronous modes. Furthermor
asynchronously sharing the underlying executors intelligently with writers. Letβs take a look at these services.
#### Clustering
-The [clustering](clustering) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into
+The [clustering](clustering.md) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into
larger ones for optimal file size management. It's fully integrated with other timeline actions like cleaning and compaction, enabling smart optimizations such as avoiding
compaction for File Groups undergoing clustering, thereby saving on I/O.
#### Compaction
-Hudi's [compaction](compaction) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows
+Hudi's [compaction](compaction.md) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows
concurrent writes to the same File Group, enabled by Hudi's file grouping and flexible log merging. This facilitates non-blocking execution of deletes even during concurrent
record updates.
@@ -107,11 +107,11 @@ while also allowing sufficient time for long running batch jobs (e.g Hive ETLs)
#### Indexing
Hudi's scalable metadata table contains auxiliary data about the table. This subsystem encompasses various indices, including files, column_stats, and bloom_filters,
facilitating efficient record location and data skipping. Balancing write throughput with index updates presents a fundamental challenge, as traditional indexing methods,
-like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchronous [metadata indexing](metadata_indexing),
+like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchronous [metadata indexing](metadata_indexing.md),
enabling the creation of various indices without impeding writes. This approach not only improves write latency but also minimizes resource waste by reducing contention between writing and indexing activities.
### Concurrency Control
-[Concurrency control](concurrency_control) defines how different writers/readers/table services coordinate access to the table. Hudi uses monotonically increasing time to sequence and order various
+[Concurrency control](concurrency_control.md) defines how different writers/readers/table services coordinate access to the table. Hudi uses monotonically increasing time to sequence and order various
changes to table state. Much like databases, Hudi take an approach of clearly differentiating between writers (responsible for upserts/deletes), table services
(focusing on storage optimization and bookkeeping), and readers (for query execution). Hudi provides snapshot isolation, offering a consistent view of the table across
these different operations. It employs lock-free, non-blocking MVCC for concurrency between writers and table-services, as well as between different table services, and
@@ -154,12 +154,12 @@ integration with engines written in C/C++.
<p align = "center">Figure: Various platform services in Hudi</p>
Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers.
-Services, like [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
+Services, like [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (including Confluent), and
deduplication of data. Hudi Streamer also offers features for backfills, one-off runs, and continuous mode operation with Spark/Flink streaming writers. Additionally,
-Hudi provides tools for [snapshotting](snapshot_exporter) and incrementally [exporting](snapshot_exporter#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback) for analytics or
+Hudi provides tools for [snapshotting](snapshot_exporter.md) and incrementally [exporting](snapshot_exporter.md#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback.md) for analytics or
workflow management, enhancing the deployment of production-grade incremental pipelines. Apart from these services, Hudi also provides broad support for different
-catalogs such as [Hive Metastore](syncing_metastore), [AWS Glue](syncing_aws_glue_data_catalog/), [Google BigQuery](gcp_bigquery), [DataHub](syncing_datahub), etc. that allows syncing of Hudi tables to be queried by
+catalogs such as [Hive Metastore](syncing_metastore.md), [AWS Glue](syncing_aws_glue_data_catalog.md), [Google BigQuery](gcp_bigquery.md), [DataHub](syncing_datahub.md), etc. that allows syncing of Hudi tables to be queried by
interactive engines such as Trino and Presto.
### Metaserver*
diff --git a/website/versioned_docs/version-1.0.1/indexes.md b/website/versioned_docs/version-1.0.1/indexes.md
index 39345053e3fe..65be944cddf5 100644
--- a/website/versioned_docs/version-1.0.1/indexes.md
+++ b/website/versioned_docs/version-1.0.1/indexes.md
@@ -19,8 +19,8 @@ Only clustering or cross-partition updates that are implemented as deletes + ins
file group at any completed instant on the timeline.
## Need for indexing
-For [Copy-On-Write tables](table_types#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](table_types#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
only against updates for records that are part of that base file.
@@ -28,7 +28,7 @@ only against updates for records that are part of that base file.
In contrast,
- Designs without an indexing component (e.g: [Apache Hive/Apache Iceberg](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)) end up having to merge all the base files against all incoming updates/delete records
- (10-100x more [read amplification](table_types#comparison)).
+ (10-100x more [read amplification](table_types.md#comparison)).
- Designs that implement heavily write-optimized OLTP data structures like LSM trees do not require an indexing component. But they perform poorly scan heavy workloads
against cloud storage making them unsuitable for serving analytical queries.
@@ -42,8 +42,8 @@ implemented by enhancing the metadata table with the flexibility to extend to ne
along with an [asynchronous index](https://hudi.apache.org/docs/metadata_indexing/#setup-async-indexing) building
Hudi supports a multi-modal index by augmenting the metadata table with the capability to incorporate new types of indexes, complemented by an
-asynchronous mechanism for [index construction](metadata_indexing). This enhancement supports a range of indexes within
-the [metadata table](metadata#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
+asynchronous mechanism for [index construction](metadata_indexing.md). This enhancement supports a range of indexes within
+the [metadata table](metadata.md#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
<p align = "center">Figure: Indexes in Hudi</p>
@@ -90,7 +90,7 @@ Following are configurations that control enabling record index building and mai
An [expression index](https://github.com/apache/hudi/blob/3789840be3d041cbcfc6b24786740210e4e6d6ac/rfc/rfc-63/rfc-63.md) is an index on a function of a column. If a query has a predicate on a function of a column, the expression index can
be used to speed up the query. Expression index is stored in *expr_index_* prefixed partitions (one for each
expression index) under metadata table. Expression index can be created using SQL syntax. Please checkout SQL DDL
-docs [here](sql_ddl#create-expression-index) for more details.
+docs [here](sql_ddl.md#create-expression-index) for more details.
Following are configurations that control enabling expression index building and maintenance on the writer.
diff --git a/website/versioned_docs/version-1.0.1/ingestion_flink.md b/website/versioned_docs/version-1.0.1/ingestion_flink.md
index 8176ff1e5ac6..8ec300910490 100644
--- a/website/versioned_docs/version-1.0.1/ingestion_flink.md
+++ b/website/versioned_docs/version-1.0.1/ingestion_flink.md
@@ -37,7 +37,7 @@ frequent `file handle` switching.
:::note
The parallelism of `bulk_insert` is specified by `write.tasks`. The parallelism will affect the number of small files.
In theory, the parallelism of `bulk_insert` is the number of `bucket`s (In particular, when each bucket writes to maximum file size, it
-will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks).
+will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks).
:::
#### Options
@@ -45,7 +45,7 @@ will rollover to the new file handle. Finally, `the number of files` >= [`write.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.operation` | `true` | `upsert` | Setting as `bulk_insert` to open this function |
-| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks) |
+| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks) |
| `write.bulk_insert.shuffle_input` | `false` | `true` | Whether to shuffle data according to the input field before writing. Enabling this option will reduce the number of small files, but there may be a risk of data skew |
| `write.bulk_insert.sort_input` | `false` | `true` | Whether to sort data according to the input field before writing. Enabling this option will reduce the number of small files when a write task writes multiple partitions |
| `write.sort.memory` | `false` | `128` | Available managed memory of sort operator. default `128` MB |
diff --git a/website/versioned_docs/version-1.0.1/metadata.md b/website/versioned_docs/version-1.0.1/metadata.md
index a9df1ffc82d9..d50d3999e163 100644
--- a/website/versioned_docs/version-1.0.1/metadata.md
+++ b/website/versioned_docs/version-1.0.1/metadata.md
@@ -62,12 +62,12 @@ Following are the different types of metadata currently supported.
even without examining column statistics at the file level. The partition stats index is stored in *partition_stats* partition in the metadata table.
Partition stats index can be enabled using the following configs (note it is required to specify the columns for which stats should be aggregated).
-To try out these features, refer to the [SQL guide](sql_ddl#create-partition-stats-index).
+To try out these features, refer to the [SQL guide](sql_ddl.md#create-partition-stats-index).
## Metadata Tracking on Writers
Following are based basic configs that are needed to enable metadata tracking. For advanced configs please refer
-[here](configurations#Metadata-Configs).
+[here](configurations.md#Metadata-Configs).
| Config Name | Default | Description |
|----------------------------------------------|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -118,10 +118,10 @@ corresponding configs across Spark and Flink readers.
To ensure that metadata table stays up to date and table metadata is tracked safely across concurrent write and
table operations, there are some additional considerations. If async table services are enabled for the table (i.e. running a separate compaction (`HoodieCompactor`) or
clustering (`HoodieClusteringJob`) job), even with just a single writer, lock providers
-must be configured. Please refer to [concurrency control](concurrency_control) for more details.
+must be configured. Please refer to [concurrency control](concurrency_control.md) for more details.
Before enabling metadata table for the first time, all writers on the same table must and table services must be stopped.
-If your current deployment model is [multi-writer](concurrency_control#full-on-multi-writer--async-table-services) along with a lock
+If your current deployment model is [multi-writer](concurrency_control.md#full-on-multi-writer--async-table-services) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only a subset of writers or table services is unsafe and can lead to loss of data. So, please ensure you enable
diff --git a/website/versioned_docs/version-1.0.1/metadata_indexing.md b/website/versioned_docs/version-1.0.1/metadata_indexing.md
index 79b08b3583a4..3a9d9de93fd3 100644
--- a/website/versioned_docs/version-1.0.1/metadata_indexing.md
+++ b/website/versioned_docs/version-1.0.1/metadata_indexing.md
@@ -5,7 +5,7 @@ toc: true
last_modified_at:
---
-Hudi maintains a scalable [metadata](metadata) that has some auxiliary data about the table.
+Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data about the table.
The [pluggable indexing subsystem](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi)
of Hudi depends on the metadata table. Different types of index, from `files` index for locating records efficiently
to `column_stats` index for data skipping, are part of the metadata table. A fundamental tradeoff in any data system
@@ -35,7 +35,7 @@ For more information on these indexes please refer [metadata section](metadata/#
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -54,7 +54,7 @@ CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(
CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
```
-For more information on index creation using SQL refer [SQL DDL](sql_ddl#create-index)
+For more information on index creation using SQL refer [SQL DDL](sql_ddl.md#create-index)
## Index Creation Using Datasource
@@ -129,7 +129,7 @@ In the example we will have continuous writing using Hudi Streamer and also crea
in example is done using HoodieIndexer so that schedule and execute phases are clearly visible for indexing. The asynchronous
configurations can be used with Datasource and SQL based configs to create index as well.
-First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) which will continuously write data
+First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) which will continuously write data
from raw parquet to Hudi table. We used the widely available [NY Taxi dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/), whose setup details are as below:
<details>
<summary>Ingestion write config</summary>
@@ -183,7 +183,7 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](indexes#multi-modal-indexing) along with [configs](metadata#metadata-tracking-on-writers)
+metadata table can be explored [here](indexes.md#multi-modal-indexing) along with [configs](metadata.md#metadata-tracking-on-writers)
to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
diff --git a/website/versioned_docs/version-1.0.1/metrics.md b/website/versioned_docs/version-1.0.1/metrics.md
index c6f2833f5ada..c389689716be 100644
--- a/website/versioned_docs/version-1.0.1/metrics.md
+++ b/website/versioned_docs/version-1.0.1/metrics.md
@@ -6,7 +6,7 @@ toc: true
last_modified_at: 2020-06-20T15:59:57-04:00
---
-In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations#METRICS).
+In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations.md#METRICS).
## MetricsReporter
@@ -17,7 +17,7 @@ MetricsReporter provides APIs for reporting `HoodieMetrics` to user-specified ba
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
#### Configurations
-The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Jmx).
+The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Jmx).
```properties
hoodie.metrics.on=true
@@ -37,7 +37,7 @@ As configured above, JmxMetricsReporter will started JMX server on port 4001. We
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send `HoodieMetrics` to it.
#### Configurations
-The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations#Metrics-Configurations-for-Graphite).
+The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Graphite).
```properties
hoodie.metrics.on=true
@@ -58,7 +58,7 @@ DatadogMetricsReporter is an implementation of Datadog reporter.
A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
#### Configurations
-The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Datadog-reporter).
+The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Datadog-reporter).
```properties
hoodie.metrics.on=true
@@ -138,7 +138,7 @@ tuned are in the `HoodieMetricsCloudWatchConfig` class.
Allows users to define a custom metrics reporter.
#### Configurations
-The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations).
+The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations).
```properties
hoodie.metrics.on=true
diff --git a/website/versioned_docs/version-1.0.1/migration_guide.md b/website/versioned_docs/version-1.0.1/migration_guide.md
index 99bafee92a75..64d87622183b 100644
--- a/website/versioned_docs/version-1.0.1/migration_guide.md
+++ b/website/versioned_docs/version-1.0.1/migration_guide.md
@@ -40,7 +40,7 @@ There are a few options when choosing this approach.
#### Using Hudi Streamer
-Use the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool. HoodieStreamer supports bootstrap with
+Use the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool. HoodieStreamer supports bootstrap with
--run-bootstrap command line option. There are two types of bootstrap, METADATA_ONLY and FULL_RECORD. METADATA_ONLY will
generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will
perform a full copy/rewrite of the data as a Hudi table. Additionally, once can choose selective partitions using regex
@@ -89,7 +89,7 @@ Refer to [Bootstrap procedure](https://hudi.apache.org/docs/next/procedures#boot
#### Using Hudi CLI
Write your own custom logic of how to load an existing table into a Hudi managed one. Please read about the RDD API
-[here](quick-start-guide). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
+[here](quick-start-guide.md). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hudi-cli && ./hudi-cli.sh`.
```java
diff --git a/website/versioned_docs/version-1.0.1/overview.mdx b/website/versioned_docs/version-1.0.1/overview.mdx
index 8f4e35c7fde7..fe320b347e16 100644
--- a/website/versioned_docs/version-1.0.1/overview.mdx
+++ b/website/versioned_docs/version-1.0.1/overview.mdx
@@ -17,14 +17,14 @@ Apache Hudi (pronounced "hoodie") pioneered the concept of "[transactional data
the data lakehouse architecture. Today, Hudi has grown into an [open data lakehouse platform](/blog/2021/07/21/streaming-data-lake-platform), with a open table format purpose-built for high performance writes on
incremental data pipelines and fast query performance due to comprehensive table optimizations.
-Hudi brings core database functionality directly to a data lake - [tables](sql_ddl),
-[transactions](timeline), [efficient upserts/deletes](write_operations), [advanced indexes](indexes),
-[ingestion services](hoodie_streaming_ingestion), data [clustering](clustering)/[compaction](compaction) optimizations,
-and [concurrency control](concurrency_control) all while keeping your data in open file formats. Not only is Apache Hudi great for streaming workloads,
-but it also allows you to create efficient incremental batch pipelines. Apache Hudi can easily be used on any [cloud storage platform](cloud).
+Hudi brings core database functionality directly to a data lake - [tables](sql_ddl.md),
+[transactions](timeline.md), [efficient upserts/deletes](write_operations.md), [advanced indexes](indexes.md),
+[ingestion services](hoodie_streaming_ingestion.md), data [clustering](clustering.md)/[compaction](compaction.md) optimizations,
+and [concurrency control](concurrency_control.md) all while keeping your data in open file formats. Not only is Apache Hudi great for streaming workloads,
+but it also allows you to create efficient incremental batch pipelines. Apache Hudi can easily be used on any [cloud storage platform](cloud.md).
Hudiβs advanced performance optimizations, make analytical queries/pipelines faster with any of the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.
-Read the docs for more [use case descriptions](use_cases) and check out [who's using Hudi](/powered-by), to see how some of the
+Read the docs for more [use case descriptions](use_cases.md) and check out [who's using Hudi](/powered-by), to see how some of the
largest data lakes in the world including [Uber](https://www.uber.com/en-IN/blog/uber-big-data-platform/), [Amazon](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/),
[ByteDance](http://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance),
[Robinhood](https://s.apache.org/hudi-robinhood-talk) and more are transforming their production data lakes with Hudi.
@@ -35,10 +35,10 @@ expands the use of Apache Hudi for a diverse range of use cases in the non-JVM e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
-- [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
-- [Hudi File Layout](storage_layouts) - How the files are laid out on storage
-- [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
-- [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+- [Hudi File Layout](storage_layouts.md) - How the files are laid out on storage
+- [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+- [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Design & Concepts" section of the docs.
@@ -48,11 +48,11 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:
-- [Spark Quick Start Guide](quick-start-guide) β if you primarily use Apache Spark
-- [Flink Quick Start Guide](flink-quick-start-guide) β if you primarily use Apache Flink
-- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide) - if you primarily use Python or Rust
+- [Spark Quick Start Guide](quick-start-guide.md) β if you primarily use Apache Spark
+- [Flink Quick Start Guide](flink-quick-start-guide.md) β if you primarily use Apache Flink
+- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide.md) - if you primarily use Python or Rust
-If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo)
+If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo.md)
## Connect With The Community
Apache Hudi is community-focused and community-led and welcomes new-comers with open arms. Leverage the following
diff --git a/website/versioned_docs/version-1.0.1/performance.md b/website/versioned_docs/version-1.0.1/performance.md
index 95e505392baa..0eabc8deab23 100644
--- a/website/versioned_docs/version-1.0.1/performance.md
+++ b/website/versioned_docs/version-1.0.1/performance.md
@@ -16,12 +16,12 @@ Here are some ways to efficiently manage the storage of your Hudi tables.
- The [small file handling feature](configurations/#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](configurations#hoodieparquetcompressionratio),
+- Cleaner can be [configured](configurations.md#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](configurations.md#hoodieparquetmaxfilesize), [log files](configurations.md#hoodielogfilemaxsize) & expected [compression ratio](configurations.md#hoodieparquetcompressionratio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
-- Intelligently tuning the [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
+- Intelligently tuning the [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be changed without re-clustering the table. Writes will simply expand given file groups with new updates/inserts as explained before.
-- For workloads with heavy updates, the [merge-on-read table](concepts#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
+- For workloads with heavy updates, the [merge-on-read table](concepts.md#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
## Performance Optimizations
@@ -119,11 +119,11 @@ For very large tables (1Tb+, 10s of 1000s of files), Data skipping could
To unlock the power of Data Skipping you will need to
-1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
+1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing.md)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
2. Enable Data Skipping in your queries, using `hoodie.metadata.index.column.stats.enable=true` (to enable Column Stats Index being populated on the write path, disabled by default)
:::note
-If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
+If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing.md) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
:::
To enable Data Skipping in your queries make sure to set following properties to `true` (on the read path):
diff --git a/website/versioned_docs/version-1.0.1/precommit_validator.md b/website/versioned_docs/version-1.0.1/precommit_validator.md
index f810b36f9b53..fe0a0dd77605 100644
--- a/website/versioned_docs/version-1.0.1/precommit_validator.md
+++ b/website/versioned_docs/version-1.0.1/precommit_validator.md
@@ -8,7 +8,7 @@ Data quality refers to the overall accuracy, completeness, consistency, and vali
Hudi offers **Pre-Commit Validators** that allow you to ensure that your data meets certain data quality expectations as you are writing with Hudi Streamer or Spark Datasource writers.
:::note
-Pre-commit validators are skipped when using the [BULK_INSERT](write_operations#bulk_insert) write operation type.
+Pre-commit validators are skipped when using the [BULK_INSERT](write_operations.md#bulk_insert) write operation type.
:::
Multiple class names can be separated by `,` delimiter.
@@ -105,7 +105,7 @@ void validateRecordsBeforeAndAfter(Dataset<Row> before,
```
## Additional Monitoring with Notifications
-Hudi offers a [commit notification service](platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
+Hudi offers a [commit notification service](platform_services_post_commit_callback.md) that can be configured to trigger notifications about write commits.
The commit notification service can be combined with pre-commit validators to send a notification when a commit fails a validation. This is possible by passing details about the validation as a custom value to the HTTP endpoint.
diff --git a/website/versioned_docs/version-1.0.1/procedures.md b/website/versioned_docs/version-1.0.1/procedures.md
index 6dae3a349b81..b7eb6e614352 100644
--- a/website/versioned_docs/version-1.0.1/procedures.md
+++ b/website/versioned_docs/version-1.0.1/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations.md#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations.md#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations.md#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations.md#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1535,11 +1535,11 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
-| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
-| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [clean_policy](configurations.md#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum qu [...]
+| [retain_commits](configurations.md#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations.md#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of [...]
+| [file_versions_retained](configurations.md#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations.md#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
@@ -1635,12 +1635,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations.md#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations.md#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations.md#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations.md#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations.md#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations.md#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1850,18 +1850,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations.md#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations.md#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations.md#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
+| [parallelism](configurations.md#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-1.0.1/python-rust-quick-start-guide.md b/website/versioned_docs/version-1.0.1/python-rust-quick-start-guide.md
index 3cabe6f1726e..fcd31e544200 100644
--- a/website/versioned_docs/version-1.0.1/python-rust-quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.1/python-rust-quick-start-guide.md
@@ -22,7 +22,7 @@ cargo add hudi
> [!NOTE]
> These examples expect a Hudi table exists at `/tmp/trips_table`, created using
-> the [quick start guide](quick-start-guide).
+> the [quick start guide](quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-1.0.1/querying_data.md b/website/versioned_docs/version-1.0.1/querying_data.md
index 61f3914fd761..2bbc335724bf 100644
--- a/website/versioned_docs/version-1.0.1/querying_data.md
+++ b/website/versioned_docs/version-1.0.1/querying_data.md
@@ -7,10 +7,10 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl.md), [SQL DML](sql_dml.md), [SQL Queries](sql_queries.md) and [Procedures](procedures.md) for the latest documentation.
:::
-Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts#query-types).
+Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts.md#query-types).
Once the table is synced to the Hive metastore, it provides external Hive tables backed by Hudi's custom inputformats. Once the proper hudi
bundle has been installed, the table can be queried by popular query engines like Hive, Spark SQL, Flink, Trino and PrestoDB.
@@ -19,7 +19,7 @@ In sections, below we will discuss specific setup to access different query type
## Spark Datasource
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
-See the [Spark Quick Start](quick-start-guide) for more examples of Spark datasource reading queries.
+See the [Spark Quick Start](quick-start-guide.md) for more examples of Spark datasource reading queries.
**Setup**
@@ -56,8 +56,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Incremental Queries](quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](configurations#SPARK_DATASOURCE) section, to view all datasource options.
+For examples, refer to [Incremental Queries](quick-start-guide.md#incremental-query) in the Spark quickstart.
+Please refer to [configurations](configurations.md#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-1.0.1/quick-start-guide.md b/website/versioned_docs/version-1.0.1/quick-start-guide.md
index c4fd789d8525..9a971b499805 100644
--- a/website/versioned_docs/version-1.0.1/quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.1/quick-start-guide.md
@@ -214,7 +214,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl.md) reference guide.
</TabItem>
@@ -258,7 +258,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -291,7 +291,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -300,7 +300,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml.md#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -412,7 +412,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml.md#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -442,7 +442,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts.md).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -546,7 +546,7 @@ Notice, instead of `UPDATE SET *`, we are updating only the `fare` column.
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data.md#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -626,7 +626,7 @@ Hudi supports indexing on columns to speed up queries. Indexes can be created on
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -1093,7 +1093,7 @@ Note that CDC queries are currently only supported on Copy-on-Write tables.
The examples thus far have showcased one of the two table types, that Hudi supports - Copy-on-Write (COW) tables. Hudi also supports
a more advanced write-optimized table type called Merge-on-Read (MOR) tables, that can balance read and write performance in a more
-flexible manner. See [table types](table_types) for more details.
+flexible manner. See [table types](table_types.md) for more details.
Any of these examples can be run on a Merge-on-Read table by simply changing the table type to MOR, while creating the table, as below.
@@ -1219,7 +1219,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations.md).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1231,8 +1231,8 @@ Hudi also uses this mechanism to support out-of-order data arrival into a table,
For e.g. using a _created_at_ timestamp field as the precombine field will prevent older versions of a record from overwriting newer ones or being exposed to queries, even
if they are written at a later commit time to the table. This is one of the key features, that makes Hudi, best suited for dealing with streaming data.
-To enable different merge semantics, Hudi supports [merge modes](record_merger). Commit time and event time based merge modes are supported out of the box.
-Users can also define their own custom merge strategies, see [here](sql_ddl#create-table-with-record-merge-mode).
+To enable different merge semantics, Hudi supports [merge modes](record_merger.md). Commit time and event time based merge modes are supported out of the box.
+Users can also define their own custom merge strategies, see [here](sql_ddl.md#create-table-with-record-merge-mode).
<Tabs
groupId="programming-language"
@@ -1295,29 +1295,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide.md).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
-For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl.md) and [Spark SQL DML](sql_dml.md) reference guides.
+For alter table commands, check out [this](sql_ddl.md#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures.md) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion.md#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes#spark-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes.md#spark-streaming) for more.
-Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data) and [streaming writes](writing_tables_streaming_writes).
+Check out more information on [modeling data in Hudi](faq_general.md#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data.md) and [streaming writes](writing_tables_streaming_writes.md).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it.
+steps [here](docker_demo.md) to get a taste for it.
diff --git a/website/versioned_docs/version-1.0.1/record_merger.md b/website/versioned_docs/version-1.0.1/record_merger.md
index 8a4f0419c553..93a149a8a991 100644
--- a/website/versioned_docs/version-1.0.1/record_merger.md
+++ b/website/versioned_docs/version-1.0.1/record_merger.md
@@ -6,7 +6,7 @@ toc_min_heading_level: 2
toc_max_heading_level: 4
---
-Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline#ordering-of-actions) section.
+Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline.md#ordering-of-actions) section.
To provide users full-fledged support for stream processing, Hudi goes all the way making the storage engine and the underlying storage format
understand how to merge changes to the same record key, that may arrive even in different order at different times. With the rise of mobile applications
and IoT, these scenarios have become the normal than an exception. For e.g. a social networking application uploading user events several hours after they happened,
@@ -54,7 +54,7 @@ With event time ordering, the merging picks the record with the highest value on
In the example above, two microservices product change records about orders at different times, that can arrive out-of-order. As color coded,
this can lead to application-level inconsistent states in the table if simply merged in commit time order like a cancelled order being re-created or
a paid order moved back to just created state expecting payment again. Event time ordering helps by ignoring older state changes that arrive late and
-avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control#non-blocking-concurrency-control-mode),
+avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control.md#non-blocking-concurrency-control-mode),
this provides a very powerful way for processing such data streams efficiently and correctly.
### CUSTOM
@@ -175,7 +175,7 @@ hoodie.datasource.write.payload.class=org.apache.hudi.common.model.EventTimeAvro
```
This is the default record payload for Flink based writing. Some use cases require merging records by event time and
thus event time plays the role of an ordering field. This payload is particularly useful in the case of late-arriving data.
-For such use cases, users need to set the [payload event time field](configurations#RECORD_PAYLOAD) configuration.
+For such use cases, users need to set the [payload event time field](configurations.md#RECORD_PAYLOAD) configuration.
#### OverwriteNonDefaultsWithLatestAvroPayload
```scala
@@ -249,7 +249,7 @@ Payload class can be specified using the below configs. For more advanced config
There are also quite a few other implementations. Developers may be interested in looking at the hierarchy of `HoodieRecordPayload` interface. For
example, [`MySqlDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/MySqlDebeziumAvroPayload.java) and [`PostgresDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/PostgresDebeziumAvroPayload.java) provides support for seamlessly applying changes
captured via Debezium for MySQL and PostgresDB. [`AWSDmsAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/AWSDmsAvroPayload.java) provides support for applying changes captured via Amazon Database Migration Service onto S3.
-For full configurations, go [here](configurations#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
+For full configurations, go [here](configurations.md#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
## Related Resources
diff --git a/website/versioned_docs/version-1.0.1/s3_hoodie.md b/website/versioned_docs/version-1.0.1/s3_hoodie.md
index 3161ea4bd284..fadcb4b01c97 100644
--- a/website/versioned_docs/version-1.0.1/s3_hoodie.md
+++ b/website/versioned_docs/version-1.0.1/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](cleaning) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](cleaning.md) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
diff --git a/website/versioned_docs/version-1.0.1/sql_ddl.md b/website/versioned_docs/version-1.0.1/sql_ddl.md
index d1a08846ecc0..565a62f480ef 100644
--- a/website/versioned_docs/version-1.0.1/sql_ddl.md
+++ b/website/versioned_docs/version-1.0.1/sql_ddl.md
@@ -75,7 +75,7 @@ should be specified as `PARTITIONED BY (dt, hh)`.
### Create table with record keys and ordering fields
-As discussed [here](quick-start-guide#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
Typically, this is also accompanied by configuring a `preCombineField` option to deal with out-of-order data and potential
duplicate records with the same key in the incoming writes.
@@ -105,7 +105,7 @@ TBLPROPERTIES (
### Create table with merge modes {#create-table-with-record-merge-mode}
-Hudi supports different [record merge modes](record_merger) to handle merge of incoming records with existing
+Hudi supports different [record merge modes](record_merger.md) to handle merge of incoming records with existing
records. To create a table with specific record merge mode, you can set `recordMergeMode` option.
```sql
@@ -127,7 +127,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode/';
With `EVENT_TIME_ORDERING`, the record with the larger event time (`precombineField`) overwrites the record with the
smaller event time on the same key, regardless of transaction's commit time. Users can set `CUSTOM` mode to provide their own
merge logic. With `CUSTOM` merge mode, you can provide a custom class that implements the merge logic. The interfaces
-to implement is explained in detail [here](record_merger#custom).
+to implement is explained in detail [here](record_merger.md#custom).
```sql
CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
@@ -147,7 +147,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion.md#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -236,7 +236,7 @@ AS SELECT * FROM parquet_table;
### Create Index
Hudi supports creating and dropping different types of indexes on a table. For more information on different
-type of indexes please refer [multi-modal indexing](indexes#multi-modal-indexing). Secondary
+type of indexes please refer [multi-modal indexing](indexes.md#multi-modal-indexing). Secondary
index, expression index and record indexes can be created using SQL create index command.
```sql
@@ -270,7 +270,7 @@ Both index and column on which the index is created can be qualified with some o
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
3. Secondary index is not supported for [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types).
:::
@@ -344,7 +344,7 @@ CREATE INDEX ts_hour ON hudi_table
:::note
1. Expression index can only be created for Spark engine using SQL. It is not supported yet with Spark DataSource API.
2. Expression index is not yet supported for [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types).
-3. Expression index is supported for unary and certain binary expressions. Please check [SQL DDL docs](sql_ddl#create-expression-index) for more details.
+3. Expression index is supported for unary and certain binary expressions. Please check [SQL DDL docs](sql_ddl.md#create-expression-index) for more details.
:::
The `expr` option is required for creating expression index, and it should be a valid Spark SQL function. Please check the syntax
@@ -577,7 +577,7 @@ Users can set table properties while creating a table. The important table prope
| Parameter Name | Default | Description |
|------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types) |
+| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types.md) |
| primaryKey | uuid | The primary key field names of the table separated by commas. Same as `hoodie.datasource.write.recordkey.field`. If this config is ignored, hudi will auto-generate primary keys. If explicitly set, primary key generation will honor user configuration. |
| preCombineField | | The pre-combine field of the table. It is used for resolving the final version of the record among multiple versions. Generally, `event time` or another similar column will be used for ordering purposes. Hudi will be able to handle out-of-order data using the preCombine field value. |
@@ -588,7 +588,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control.md#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -839,7 +839,7 @@ WITH (
### Create Table in Non-Blocking Concurrency Control Mode
-The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control#non-blocking-concurrency-control).
+The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control.md#non-blocking-concurrency-control).
```sql
-- This is a datagen source that can generate records continuously
@@ -907,7 +907,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations.md#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-1.0.1/sql_dml.md b/website/versioned_docs/version-1.0.1/sql_dml.md
index 462eafb9699e..71f49884a848 100644
--- a/website/versioned_docs/version-1.0.1/sql_dml.md
+++ b/website/versioned_docs/version-1.0.1/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
### Insert Into
@@ -29,7 +29,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -49,7 +49,7 @@ INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000
```
:::note Mapping to write operations
-Hudi offers flexibility in choosing the underlying [write operation](write_operations) of a `INSERT INTO` statement using
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
and *"upsert"* (with deduplication/merging). If a precombine field is not set, *"insert"* is chosen as the default. For a table with preCombine field set,
*"upsert"* is chosen as the default operation.
@@ -407,7 +407,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control.md#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-1.0.1/sql_queries.md b/website/versioned_docs/version-1.0.1/sql_queries.md
index 1ab1f1dc8289..aca0a0ac8ccc 100644
--- a/website/versioned_docs/version-1.0.1/sql_queries.md
+++ b/website/versioned_docs/version-1.0.1/sql_queries.md
@@ -8,11 +8,11 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-Hudi stores and organizes data on storage while providing different ways of [querying](concepts#query-types), across a wide range of query engines.
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
This page will show how to issue different queries and discuss any specific instructions for each query engine.
## Spark SQL
-The Spark [quickstart](quick-start-guide) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
@@ -90,7 +90,7 @@ show the files scanned as 1 compared to 3 files scanned without index.
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -196,7 +196,7 @@ DROP INDEX partition_stats on hudi_indexed_table;
### Snapshot Query with Event Time Ordering
-Hudi supports different [record merge modes](record_merger) for merging the records from the same key. Event
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
event time ordering merge mode.
@@ -323,7 +323,7 @@ For e.g the table has received 10 million modifications across 1 million records
see all changes in a given time window and not just the latest values.
:::
-Please refer to [configurations](basic_configurations) section for the important configuration options.
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
@@ -406,7 +406,7 @@ and the corresponding type column value.
## Flink SQL
Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
-relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide).
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-1.0.1/storage_layouts.md b/website/versioned_docs/version-1.0.1/storage_layouts.md
index 3b7911a15a1d..a88f584e3f9d 100644
--- a/website/versioned_docs/version-1.0.1/storage_layouts.md
+++ b/website/versioned_docs/version-1.0.1/storage_layouts.md
@@ -12,8 +12,8 @@ The following describes the general organization of files in storage for a Hudi
* Each slice contains a **_base file_** (parquet/orc/hfile) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) )
written by a commit that completed at a certain instant, along with set of **_log files_** (*.log.*) written by commits that completed
before the next base file's requested instant.
-* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
- file slices and [cleaning](cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+ file slices and [cleaning](cleaning.md) action gets rid of unused/older file slices to reclaim space on the file system.
* All metadata including timeline, metadata table are stored in a special `.hoodie` directory under the base path.
diff --git a/website/versioned_docs/version-1.0.1/structure.md b/website/versioned_docs/version-1.0.1/structure.md
index 31281b68f71c..9f78f89f16f5 100644
--- a/website/versioned_docs/version-1.0.1/structure.md
+++ b/website/versioned_docs/version-1.0.1/structure.md
@@ -17,4 +17,4 @@ Hudi (pronounced βHoodieβ) ingests & manages storage of large analytical tab
By carefully managing how data is laid out in storage & how itβs exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [PrestoDB](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) [...]
-Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide) for a demo.
+Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide.md) for a demo.
diff --git a/website/versioned_docs/version-1.0.1/syncing_aws_glue_data_catalog.md b/website/versioned_docs/version-1.0.1/syncing_aws_glue_data_catalog.md
index b6f6c82a6c54..35f43a8af472 100644
--- a/website/versioned_docs/version-1.0.1/syncing_aws_glue_data_catalog.md
+++ b/website/versioned_docs/version-1.0.1/syncing_aws_glue_data_catalog.md
@@ -10,7 +10,7 @@ and send them to AWS Glue.
## Configurations
Most of the configurations for `AwsGlueCatalogSyncTool` are shared with `HiveSyncTool`. The example showed in
-[Sync to Hive Metastore](syncing_metastore) can be used as is for sync with Glue Data Catalog, provided that the hive metastore
+[Sync to Hive Metastore](syncing_metastore.md) can be used as is for sync with Glue Data Catalog, provided that the hive metastore
URL (either JDBC or thrift URI) can proxied to Glue Data Catalog, which is usually done within AWS EMR or Glue job environment.
For Hudi streamer, users can set
diff --git a/website/versioned_docs/version-1.0.1/syncing_metastore.md b/website/versioned_docs/version-1.0.1/syncing_metastore.md
index d29eea66f8b0..f7c15692f251 100644
--- a/website/versioned_docs/version-1.0.1/syncing_metastore.md
+++ b/website/versioned_docs/version-1.0.1/syncing_metastore.md
@@ -124,7 +124,7 @@ will be inferred and set automatically.
## Hive Sync Tool
-Writing data with [DataSource](writing_data) writer or [HoodieStreamer](hoodie_streaming_ingestion) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
+Writing data with [DataSource](writing_data.md) writer or [HoodieStreamer](hoodie_streaming_ingestion.md) supports syncing of the table's latest schema to Hive metastore, such that queries can pick up new columns and partitions.
In case, it's preferable to run this from commandline or in an independent jvm, Hudi provides a `HiveSyncTool`, which can be invoked as below,
once you have built the hudi-hive module. Following is how we sync the above Datasource Writer written table to Hive metastore.
diff --git a/website/versioned_docs/version-1.0.1/use_cases.md b/website/versioned_docs/version-1.0.1/use_cases.md
index 2194e3108e60..251df6187611 100644
--- a/website/versioned_docs/version-1.0.1/use_cases.md
+++ b/website/versioned_docs/version-1.0.1/use_cases.md
@@ -19,7 +19,7 @@ it is unfortunately often solved in a piecemeal fashion, using a medley of inges
### Why Hudi?
- Unique design choices like Merge-On-Read tables, record-level indexes and asynchronous compaction, approach theoretical optimality for absorbing changes to tables quickly and efficiently.
-- Built-in ingestion tools on [Spark](hoodie_streaming_ingestion), [Flink](ingestion_flink) and [Kafka Connect](ingestion_kafka_connect), that let you ingest data with a single command.
+- Built-in ingestion tools on [Spark](hoodie_streaming_ingestion.md), [Flink](ingestion_flink.md) and [Kafka Connect](ingestion_kafka_connect.md), that let you ingest data with a single command.
- Support for incremental ingestion with automatic checkpoint management from streaming sources (Kafka, Pulsar, ...), Cloud storage (S3, GCS, ADLS, etc.) and even JDBC.
- Support for widely used data formats (Protobuf, Avro, JSON), file formats (parquet, orc, avro, etc.) and change log formats like [Debezium](http://debezium.io/).
- Even for scalable de-duplication for high-volume append-only streaming data, by employing bloom filter indexes and advanced data structures like interval trees for efficient range pruning.
@@ -65,10 +65,10 @@ together to build out the platform. Such an open platform is also essential for
- While open data formats help, Hudi unlocks complete freedom by also providing open compute services for ingesting, optimizing, indexing and querying data. For e.g Hudi's writers come with
a self-managing table service runtime that can maintain tables automatically in the background on each write. Often times, Hudi and your favorite open query engine is all
you need to get an open data platform up and running.
-- Examples of open services that make performance optimization or management easy include: [auto file sizing](file_sizing) to solve the "small files" problem,
- [clustering](clustering) to co-locate data next to each other, [compaction](compaction) to allow tuning of low latency ingestion + fast read queries,
- [indexing](indexes) - for faster writes/queries, Multi-Dimensional Partitioning (Z-Ordering), automatic cleanup of uncommitted data with marker mechanism,
- [auto cleaning](cleaning) to automatically removing old versions of files.
+- Examples of open services that make performance optimization or management easy include: [auto file sizing](file_sizing.md) to solve the "small files" problem,
+ [clustering](clustering.md) to co-locate data next to each other, [compaction](compaction.md) to allow tuning of low latency ingestion + fast read queries,
+ [indexing](indexes.md) - for faster writes/queries, Multi-Dimensional Partitioning (Z-Ordering), automatic cleanup of uncommitted data with marker mechanism,
+ [auto cleaning](cleaning.md) to automatically removing old versions of files.
- Hudi provides rich options for pre-sorting/loading data efficiently and then follow on with rich set of data clustering techniques to manage file sizes and data distribution within a table. In each case, Hudi provides high-degree of configurability in terms of when/how often these services are scheduled, planned and executed. For e.g. Hudi ships with a handful of common planning strategies for compaction and clustering.
- Along with compatibility with other open table formats like [Apache Iceberg](https://iceberg.apache.org/)/[Delta Lake](https://delta.io/), and catalog sync services to various data catalogs, Hudi is one of the most open choices for your data foundation.
diff --git a/website/versioned_docs/version-1.0.1/write_operations.md b/website/versioned_docs/version-1.0.1/write_operations.md
index a3a1311ac8b6..d2e410da2a73 100644
--- a/website/versioned_docs/version-1.0.1/write_operations.md
+++ b/website/versioned_docs/version-1.0.1/write_operations.md
@@ -119,17 +119,17 @@ Here are the basic configs relevant to the write operations types mentioned abov
## Write path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](configurations#hoodiecombinebeforeinsert) : First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](indexes) : Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](file_sizing): Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](storage_layouts): We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
+1. [Deduping](configurations.md#hoodiecombinebeforeinsert) : First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
+2. [Index Lookup](indexes.md) : Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
+3. [File Sizing](file_sizing.md): Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
+4. [Partitioning](storage_layouts.md): We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O :Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](indexes): Now that the write is performed, we will go back and update the index.
-7. Commit: Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback) can be configured.)
-8. [Clean](cleaning) (if needed): Following the commit, cleaning is invoked if needed.
-9. [Compaction](compaction): If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
-10. Archive : Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
+6. Update [Index](indexes.md): Now that the write is performed, we will go back and update the index.
+7. Commit: Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback.md) can be configured.)
+8. [Clean](cleaning.md) (if needed): Following the commit, cleaning is invoked if needed.
+9. [Compaction](compaction.md): If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
+10. Archive : Lastly, we perform an archival step which moves old [timeline](timeline.md) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-1.0.1/writing_data.md b/website/versioned_docs/version-1.0.1/writing_data.md
index edf26430a06f..965903d04f14 100644
--- a/website/versioned_docs/version-1.0.1/writing_data.md
+++ b/website/versioned_docs/version-1.0.1/writing_data.md
@@ -28,15 +28,15 @@ Default value: `"partitionpath"`<br/>
**PRECOMBINE_FIELD**: When two records within the same batch have the same key value, the record with the largest value from the field specified will be choosen. If you are using default payload of OverwriteWithLatestAvroPayload for HoodieRecordPayload (`WRITE_PAYLOAD_CLASS`), an incoming record will always takes precendence compared to the one in storage ignoring this `PRECOMBINE_FIELD_OPT_KEY`. <br/>
Default value: `"ts"`<br/>
-**OPERATION**: The [write operations](write_operations) to use.<br/>
+**OPERATION**: The [write operations](write_operations.md) to use.<br/>
Available values:<br/>
`"upsert"` (default), `"bulk_insert"`, `"insert"`, `"delete"`
-**TABLE_TYPE**: The [type of table](concepts#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
+**TABLE_TYPE**: The [type of table](concepts.md#table-types) to write to. Note: After the initial creation of a table, this value must stay consistent when writing to (updating) the table using the Spark `SaveMode.Append` mode.<br/>
Available values:<br/>
-[`COW_TABLE_TYPE_OPT_VAL`](concepts#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts#merge-on-read-table)
+[`COW_TABLE_TYPE_OPT_VAL`](concepts.md#copy-on-write-table) (default), [`MOR_TABLE_TYPE_OPT_VAL`](concepts.md#merge-on-read-table)
-**KEYGENERATOR_CLASS_NAME**: Refer to [Key Generation](key_generation) section below.
+**KEYGENERATOR_CLASS_NAME**: Refer to [Key Generation](key_generation.md) section below.
Example:
@@ -84,9 +84,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -120,9 +120,9 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
-and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion).
+and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](hoodie_streaming_ingestion.md).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](write_operations.md)
:::
</TabItem>
@@ -421,7 +421,7 @@ Then any record you want to delete you can mark `_hoodie_is_deleted` as true:
Following is an example of how to use `optimistic_concurrency_control` via Spark DataSource API.
-Read more in-depth details about concurrency control in the [concurrency control concepts](concurrency_control) section.
+Read more in-depth details about concurrency control in the [concurrency control concepts](concurrency_control.md) section.
```java
inputDF.write.format("hudi")
diff --git a/website/versioned_docs/version-1.0.2/basic_configurations.md b/website/versioned_docs/version-1.0.2/basic_configurations.md
index eccefb1d0e90..9afdb2db344a 100644
--- a/website/versioned_docs/version-1.0.2/basic_configurations.md
+++ b/website/versioned_docs/version-1.0.2/basic_configurations.md
@@ -1,11 +1,11 @@
---
title: Basic Configurations
-summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+summary: This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
last_modified_at: 2025-05-02T16:59:00.595581
---
-This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations) page.
+This page covers the basic configurations you may use to write/read Hudi tables. This page only features a subset of the most frequently used configurations. For a full list of all configs, please visit the [All Configurations](configurations.md) page.
- [**Hudi Table Config**](#TABLE_CONFIG): Basic Hudi Table configuration parameters.
- [**Spark Datasource Configs**](#SPARK_DATASOURCE): These configs control the Hudi Spark Datasource, providing ability to define keys/partitioning, pick out the write operation, specify how to merge records or choosing query type to read.
diff --git a/website/versioned_docs/version-1.0.2/cleaning.md b/website/versioned_docs/version-1.0.2/cleaning.md
index c0735b240ad2..704bb27428f1 100644
--- a/website/versioned_docs/version-1.0.2/cleaning.md
+++ b/website/versioned_docs/version-1.0.2/cleaning.md
@@ -133,7 +133,7 @@ spark-submit --master local \
Note: The parallelism takes the min value of number of partitions to clean and `hoodie.cleaner.parallelism`.
#### CLI
-You can also use [Hudi CLI](cli) to run Hoodie Cleaner.
+You can also use [Hudi CLI](cli.md) to run Hoodie Cleaner.
CLI provides the below commands for cleaner service:
- `cleans show`
diff --git a/website/versioned_docs/version-1.0.2/cli.md b/website/versioned_docs/version-1.0.2/cli.md
index f82c330f7295..422a299e8583 100644
--- a/website/versioned_docs/version-1.0.2/cli.md
+++ b/website/versioned_docs/version-1.0.2/cli.md
@@ -582,7 +582,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery.md)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-1.0.2/cloud.md b/website/versioned_docs/version-1.0.2/cloud.md
index 123abd5e6bea..a3477b551bee 100644
--- a/website/versioned_docs/version-1.0.2/cloud.md
+++ b/website/versioned_docs/version-1.0.2/cloud.md
@@ -11,23 +11,23 @@ last_modified_at: 2021-10-12T10:50:00+08:00
Immaterial of whether RDD/WriteClient APIs or Datasource is used, the following information helps configure access
to cloud stores.
-* [AWS S3](s3_hoodie) <br/>
+* [AWS S3](s3_hoodie.md) <br/>
Configurations required for S3 and Hudi co-operability.
-* [Google Cloud Storage](gcs_hoodie) <br/>
+* [Google Cloud Storage](gcs_hoodie.md) <br/>
Configurations required for GCS and Hudi co-operability.
-* [Alibaba Cloud OSS](oss_hoodie) <br/>
+* [Alibaba Cloud OSS](oss_hoodie.md) <br/>
Configurations required for OSS and Hudi co-operability.
-* [Microsoft Azure](azure_hoodie) <br/>
+* [Microsoft Azure](azure_hoodie.md) <br/>
Configurations required for Azure and Hudi co-operability.
-* [Tencent Cloud Object Storage](cos_hoodie) <br/>
+* [Tencent Cloud Object Storage](cos_hoodie.md) <br/>
Configurations required for COS and Hudi co-operability.
-* [IBM Cloud Object Storage](ibm_cos_hoodie) <br/>
+* [IBM Cloud Object Storage](ibm_cos_hoodie.md) <br/>
Configurations required for IBM Cloud Object Storage and Hudi co-operability.
-* [Baidu Cloud Object Storage](bos_hoodie) <br/>
+* [Baidu Cloud Object Storage](bos_hoodie.md) <br/>
Configurations required for BOS and Hudi co-operability.
-* [JuiceFS](jfs_hoodie) <br/>
+* [JuiceFS](jfs_hoodie.md) <br/>
Configurations required for JuiceFS and Hudi co-operability.
-* [Oracle Cloud Infrastructure](oci_hoodie) <br/>
+* [Oracle Cloud Infrastructure](oci_hoodie.md) <br/>
Configurations required for OCI and Hudi co-operability.
:::note
diff --git a/website/versioned_docs/version-1.0.2/clustering.md b/website/versioned_docs/version-1.0.2/clustering.md
index 74c58bdd0625..c3080a48f6aa 100644
--- a/website/versioned_docs/version-1.0.2/clustering.md
+++ b/website/versioned_docs/version-1.0.2/clustering.md
@@ -13,7 +13,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## How is compaction different from clustering?
Hudi is modeled like a log-structured storage engine with multiple versions of the data.
-Particularly, [Merge-On-Read](table_types#merge-on-read-table)
+Particularly, [Merge-On-Read](table_types.md#merge-on-read-table)
tables in Hudi store data using a combination of base file in columnar format and row-based delta logs that contain
updates. Compaction is a way to merge the delta logs with base files to produce the latest file slices with the most
recent snapshot of data. Compaction helps to keep the query performance in check (larger delta log files would incur
@@ -93,7 +93,7 @@ update strategy.
This strategy comes into play while creating clustering plan. It helps to decide what file groups should be clustered
and how many output file groups should the clustering produce. Note that these strategies are easily pluggable using the
-config [hoodie.clustering.plan.strategy.class](configurations#hoodieclusteringplanstrategyclass).
+config [hoodie.clustering.plan.strategy.class](configurations.md#hoodieclusteringplanstrategyclass).
Different plan strategies are as follows:
diff --git a/website/versioned_docs/version-1.0.2/compaction.md b/website/versioned_docs/version-1.0.2/compaction.md
index c4d3b45e89d3..22a0562485a9 100644
--- a/website/versioned_docs/version-1.0.2/compaction.md
+++ b/website/versioned_docs/version-1.0.2/compaction.md
@@ -13,7 +13,7 @@ not applicable to Copy On Write(COW) tables and only applies to MOR tables.
### Why MOR tables need compaction?
To understand the significance of compaction in MOR tables, it is helpful to understand the MOR table layout first. In Hudi,
-data is organized in terms of [file groups](storage_layouts/). Each file group in a MOR table
+data is organized in terms of [file groups](storage_layouts.md). Each file group in a MOR table
consists of a base file and one or more log files. Typically, during writes, inserts are stored in the base file, and updates
are appended to log files.
@@ -182,7 +182,7 @@ This model may need a lock provider **if** metadata table is enabled.
:::
#### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](cli.md#compactions)
The compactor utility allows to do scheduling and execution of compaction.
Example:
@@ -199,7 +199,7 @@ Note, the `instant-time` parameter is now optional for the Hudi Compactor Utilit
the spark-submit will execute the earliest scheduled compaction on the Hudi timeline.
#### Hudi CLI
-Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli#compactions)
+Hudi CLI is yet another way to execute specific compactions asynchronously. Here is an example and you can read more in the [deployment guide](cli.md#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-1.0.2/concurrency_control.md b/website/versioned_docs/version-1.0.2/concurrency_control.md
index 0a4a39d1fa5b..7af4fd085f35 100644
--- a/website/versioned_docs/version-1.0.2/concurrency_control.md
+++ b/website/versioned_docs/version-1.0.2/concurrency_control.md
@@ -8,7 +8,7 @@ last_modified_at: 2021-03-19T15:59:57-04:00
---
Concurrency control defines how different writers/readers/table services coordinate access to a Hudi table. Hudi ensures atomic writes, by way of publishing commits atomically to the timeline,
stamped with an instant time that denotes the time at which the action is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between
-writer processes that issue [write operations](write_operations) and table services that (re)write data/metadata to optimize/perform bookkeeping and
+writer processes that issue [write operations](write_operations.md) and table services that (re)write data/metadata to optimize/perform bookkeeping and
readers (that execute queries and read data).
Hudi provides
@@ -234,7 +234,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.1-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](sql_dml#non-blocking-concurrency-control-experimental).
+writers [here](sql_dml.md#non-blocking-concurrency-control-experimental).
:::note
`NON_BLOCKING_CONCURRENCY_CONTROL` between ingestion writer and table service writer is not yet supported for clustering.
diff --git a/website/versioned_docs/version-1.0.2/deployment.md b/website/versioned_docs/version-1.0.2/deployment.md
index 1e0df56da363..f617f20cac64 100644
--- a/website/versioned_docs/version-1.0.2/deployment.md
+++ b/website/versioned_docs/version-1.0.2/deployment.md
@@ -26,13 +26,13 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### Hudi Streamer
-[Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) is the standalone utility to incrementally pull upstream changes
+[Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) is the standalone utility to incrementally pull upstream changes
from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in two modes.
To use Hudi Streamer in Spark, the `hudi-utilities-slim-bundle` and Hudi Spark bundle are required, by adding
`--packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.1,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.1` to the `spark-submit` command.
- - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -133,7 +133,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Batch Writes](writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.d [...]
+As described in [Batch Writes](writing_data.md#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
@@ -260,7 +260,7 @@ hoodie.table.version=5
```
Lets try downgrading the table back to version 4. For downgrading we will need to use Hudi CLI and execute downgrade.
-For more details on downgrade, please refer documentation [here](cli#upgrade-and-downgrade-table).
+For more details on downgrade, please refer documentation [here](cli.md#upgrade-and-downgrade-table).
```shell
connect --path /tmp/hudi_table
downgrade table --toVersion 4
@@ -281,4 +281,4 @@ Currently migrating to Hudi can be done using two approaches
- **Full conversion to Hudi** : This model is suitable if you are currently bulk/full loading the table few times a day (e.g database ingestion). The full conversion of Hudi is simply a one-time step (akin to 1 run of your existing job),
which moves all of the data into the Hudi format and provides the ability to incrementally update for future writes.
-For more details, refer to the detailed [migration guide](migration_guide). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
+For more details, refer to the detailed [migration guide](migration_guide.md). In the future, we will be supporting seamless zero-copy bootstrap of existing tables with all the upsert/incremental query capabilities fully supported.
diff --git a/website/versioned_docs/version-1.0.2/faq.md b/website/versioned_docs/version-1.0.2/faq.md
index 26c3eb50d214..c53c8ebbc68c 100644
--- a/website/versioned_docs/version-1.0.2/faq.md
+++ b/website/versioned_docs/version-1.0.2/faq.md
@@ -6,10 +6,10 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](faq_general)
-- [Design & Concepts](faq_design_and_concepts)
-- [Writing Tables](faq_writing_tables)
-- [Reading Tables](faq_reading_tables)
-- [Table Services](faq_table_services)
-- [Storage](faq_storage)
-- [Integrations](faq_integrations)
+- [General](faq_general.md)
+- [Design & Concepts](faq_design_and_concepts.md)
+- [Writing Tables](faq_writing_tables.md)
+- [Reading Tables](faq_reading_tables.md)
+- [Table Services](faq_table_services.md)
+- [Storage](faq_storage.md)
+- [Integrations](faq_integrations.md)
diff --git a/website/versioned_docs/version-1.0.2/faq_design_and_concepts.md b/website/versioned_docs/version-1.0.2/faq_design_and_concepts.md
index 720f4ae8e2df..8aee10defc1a 100644
--- a/website/versioned_docs/version-1.0.2/faq_design_and_concepts.md
+++ b/website/versioned_docs/version-1.0.2/faq_design_and_concepts.md
@@ -6,14 +6,14 @@ keywords: [hudi, writing, reading]
### How does Hudi ensure atomicity?
-Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
+Hudi writers atomically move an inflight write operation to a "completed" state by writing an object/file to the [timeline](timeline.md) folder, identifying the write operation with an instant time that denotes the time the action is deemed to have occurred. This is achieved on the underlying DFS (in the case of S3/Cloud Storage, by an atomic PUT operation) and can be observed by files of the pattern `<instant>.<action>.<state>` in Hudiβs timeline.
### Does Hudi extend the Hive table layout?
Hudi is very different from Hive in important aspects described below. However, based on practical considerations, it chooses to be compatible with Hive table layout by adopting partitioning, schema evolution and being queryable through Hive query engine. Here are the key aspect where Hudi differs:
-* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
-* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users exper [...]
+* Unlike Hive, Hudi does not remove the partition columns from the data files. Hudi in fact adds record level [meta fields](/learn/tech-specs#meta-fields) including instant time, primary record key, and partition path to the data to support efficient upserts and [incremental queries/ETL](use_cases.md#efficient-data-lakes-with-incremental-processing).Β Hudi tables can be non-partitioned and the Hudi metadata table adds rich indexes on Hudi tables which are beyond simple Hive extensions.
+* Hive advocates partitioning as the main remedy for most performance-based issues. Features like partition evolution and hidden partitioning are primarily based on this Hive based principle of partitioning and aim to tackle the metadata problem partially.Β Whereas, Hudi biases to coarse-grained partitioning and emphasizes [clustering](clustering.md) for more fine-grained partitioning. Further, users can strategize and evolve the clustering asynchronously which βactuallyβ help users ex [...]
* Hudi considers partition evolution as an anti-pattern and avoids such schemes due to the inconsistent performance of queries that goes to depend on which part of the table is being queried. Hudiβs design favors consistent performance and is aware of the need to redesign to partitioning/tables to achieve the same.
### What concurrency control approaches does Hudi adopt?
@@ -32,7 +32,7 @@ In these scenarios, it might be tempting to think of data inconsistencies/data l
(examples [1](https://github.com/apache/hudi/blob/aea5bb6f0ab824247f5e3498762ad94f643a2cb6/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L76),
[2](https://github.com/apache/hudi/blame/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java#L173)) in incremental queries to ensure that no data
is served beyond the point there is an inflight instant in its timeline, so no data loss or drop happens. This detection is made possible because Hudi writes first request a transaction on the timeline, before planning/executing
-the write, as explained in the [timeline](timeline#states) section.
+the write, as explained in the [timeline](timeline.md#states) section.
In this case, on seeing C1βs inflight commit (publish to timeline is atomic), C2 data (which is > C1 in the timeline) is not served until C1 inflight transitions to a terminal state such as completed or marked as failed.
This [test](https://github.com/apache/hudi/blob/master/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java#L137) demonstrates how Hudi incremental source stops proceeding until C1 completes.
@@ -48,7 +48,7 @@ To expand more on the long term approach, Hudi has had a proposal to streamline/
This has been delayed for a few reasons
- Large hosted query engines and users not upgrading fast enough.
-- The issues brought up - \[[1](faq_design_and_concepts#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
+- The issues brought up - \[[1](faq_design_and_concepts.md#does-hudis-use-of-wall-clock-timestamp-for-instants-pose-any-clock-skew-issues),[2](faq_design_and_concepts.md#hudis-commits-are-based-on-transaction-start-time-instead-of-completed-time-does-this-cause-data-loss-or-inconsistency-in-case-of-incremental-and-time-travel-queries)\],
relevant to this are not practically very important to users beyond good pedantic discussions,
- Wanting to do it alongside [non-blocking concurrency control](https://github.com/apache/hudi/pull/7907) in Hudi version 1.x.
diff --git a/website/versioned_docs/version-1.0.2/faq_general.md b/website/versioned_docs/version-1.0.2/faq_general.md
index cb9e0161bcb8..bfefd3b6d851 100644
--- a/website/versioned_docs/version-1.0.2/faq_general.md
+++ b/website/versioned_docs/version-1.0.2/faq_general.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### When is Hudi useful for me or my organization?
-If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
+If you are looking to quickly ingest data onto HDFS or cloud storage, Hudi provides you [tools](hoodie_streaming_ingestion.md). Also, if you have ETL/hive/spark jobs which are slow/taking up a lot of resources, Hudi can potentially help by providing an incremental approach to reading and writing data.
As an organization, Hudi can help you build an [efficient data lake](https://docs.google.com/presentation/d/1FHhsvh70ZP6xXlHdVsAI0g__B_6Mpto5KQFlZ0b8-mM/edit#slide=id.p), solving some of the most complex, low-level storage management problems, while putting data into hands of your data analysts, engineers and scientists much quicker.
@@ -37,7 +37,7 @@ Choose Copy-on-write storage if :
* You are looking for a simple alternative, that replaces your existing parquet tables without any need for real-time data.
* Your current job is rewriting entire table/partition to deal with updates, while only a few files actually change in each partition.
-* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
+* You are happy keeping things operationally simpler (no compaction etc), with the ingestion/write performance bound by the [parquet file size](configurations.md#hoodieparquetmaxfilesize) and the number of such files affected/dirtied by updates
* Your workload is fairly well-understood and does not have sudden bursts of large amount of update or inserts to older partitions. COW absorbs all the merging cost on the writer side and thus these sudden changes can clog up your ingestion and interfere with meeting normal mode ingest latency targets.
Choose merge-on-read storage if :
@@ -51,7 +51,7 @@ Immaterial of what you choose, Hudi provides
* Incremental pulls
* Ability to de-duplicate data
-Find more [here](concepts/).
+Find more [here](concepts.md).
### Is Hudi an analytical database?
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data.md) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
@@ -79,7 +79,7 @@ generate the key internally (much like RDBMS row_ids)
### How does Hudi actually store data inside a table?
-At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts/) for more details.
+At a high level, Hudi is based on MVCC design that writes data to versioned parquet/base files and log files that contain changes to the base file. All the files are stored under a partitioning scheme for the table, which closely resembles how Apache Hive tables are laid out on DFS. Please refer [here](concepts.md) for more details.
### How Hudi handles partition evolution requirements ?
diff --git a/website/versioned_docs/version-1.0.2/faq_reading_tables.md b/website/versioned_docs/version-1.0.2/faq_reading_tables.md
index 3f413b7ef1e0..e9db97dbbdd2 100644
--- a/website/versioned_docs/version-1.0.2/faq_reading_tables.md
+++ b/website/versioned_docs/version-1.0.2/faq_reading_tables.md
@@ -24,7 +24,7 @@ set hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat
### Does Hudi guarantee consistent reads? How to think about read optimized queries?
-Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
+Hudi does offer consistent reads. To read the latest snapshot of a MOR table, a user should use snapshot query. The [read-optimized queries](table_types.md#query-types) (targeted for the MOR table ONLY) are an add on benefit to provides users with a practical tradeoff of decoupling writer performance vs query performance, leveraging the fact that most queries query say the most recent data in the table.
Hudiβs read-optimized query is targeted for the MOR table only, with guidance around how compaction should be run to achieve predictable results. In the MOR table, the compaction, which runs every few commits (or βdeltacommitβ to be exact for the MOR table) by default, merges the base (parquet) file and corresponding change log files to a new base file within each file group, so that the snapshot query serving the latest data immediately after compaction reads the base files only.Β Simil [...]
diff --git a/website/versioned_docs/version-1.0.2/faq_storage.md b/website/versioned_docs/version-1.0.2/faq_storage.md
index b088b30d8ebd..2f6950fed334 100644
--- a/website/versioned_docs/version-1.0.2/faq_storage.md
+++ b/website/versioned_docs/version-1.0.2/faq_storage.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### Does Hudi support cloud storage/object stores?
-Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
+Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop FileSystem implementation and thus can read and write tables on [Cloud stores](cloud.md) (Amazon S3 or Microsoft Azure or Google Cloud Storage). Over time, Hudi has also incorporated specific design aspects that make building Hudi tables on the cloud easy, such as [consistency checks for s3](configurations.md#hoodieconsistencycheckenabled), Zero moves/renames involved for data files.
### What is the difference between copy-on-write (COW) vs merge-on-read (MOR) table types?
@@ -14,11 +14,11 @@ Yes. Generally speaking, Hudi is able to provide its functionality on any Hadoop
**Merge On Read** - This storage type enables clients to ingest data quickly onto row based data format such as avro. Any new data that is written to the Hudi table using MOR table type, will write new log/delta files that internally store the data as avro encoded bytes. A compaction process (configured as inline or asynchronous) will convert log file format to columnar file format (parquet). Two different InputFormats expose 2 different views of this data, Read Optimized view exposes co [...]
-More details can be found [here](concepts/) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
+More details can be found [here](concepts.md) and also [Design And Architecture](https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture).
### How do I migrate my data to Hudi?
-Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide/), including ways to doing partial migrations.
+Hudi provides built in support for rewriting your entire table into Hudi one-time using the HDFSParquetImporter tool available from the hudi-cli . You could also do this via a simple read and write of the dataset using the Spark datasource APIs. Once migrated, writes can be performed using normal means discussed [here](faq_writing_tables.md#what-are-some-ways-to-write-a-hudi-table). This topic is discussed in detail [here](migration_guide.md), including ways to doing partial migrations.
### How to convert an existing COW table to MOR?
@@ -34,7 +34,7 @@ But manually changing it will result in checksum errors. So, we have to go via h
### How can I find the average record size in a commit?
-The `commit showpartitons` command in [HUDI CLI](cli) will show both "bytes written" and
+The `commit showpartitons` command in [HUDI CLI](cli.md) will show both "bytes written" and
"records inserted." Divide the bytes written by records inserted to find the average size. Note that this answer assumes
@@ -91,11 +91,11 @@ spark.read.parquet("your_data_set/path/to/month").limit(n) // Limit n records
.save(basePath);
```
-For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli).
+For merge on read table, you may want to also try scheduling and running compaction jobs. You can run compaction directly using spark submit on org.apache.hudi.utilities.HoodieCompactor or by using [HUDI CLI](cli.md).
### Why does maintain record level commit metadata? Isn't tracking table version at file level good enough?Β
-By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hudi [...]
+By generating a commit time ahead of time, Hudi is able to stamp each record with effectively a transaction id that it's part of that commit enabling record level change tracking. This means, that even if that file is compacted/clustered ([they mean different things in Hudi](clustering.md#how-is-compaction-different-from-clustering)) many times, in between incremental queries, we are able to [preserve history of the records](/blog/2023/05/19/hudi-metafields-demystified). Further more, Hu [...]
### Why partition fields are also stored in parquet files in addition to the partition path ?
diff --git a/website/versioned_docs/version-1.0.2/faq_table_services.md b/website/versioned_docs/version-1.0.2/faq_table_services.md
index abaa63226a8e..cdf662906cf6 100644
--- a/website/versioned_docs/version-1.0.2/faq_table_services.md
+++ b/website/versioned_docs/version-1.0.2/faq_table_services.md
@@ -6,11 +6,11 @@ keywords: [hudi, writing, reading]
### What does the Hudi cleaner do?
-The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be read [...]
+The Hudi cleaner process often runs right after a commit and deltacommit and goes about deleting old files that are no longer needed. If you are using the incremental pull feature, then ensure you configure the cleaner to [retain sufficient amount of last commits](configurations.md#hoodiecleanercommitsretained) to rewind. Another consideration is to provide sufficient time for your long running jobs to finish running. Otherwise, the cleaner could delete a file that is being or could be r [...]
### How do I run compaction for a MOR table?
-Simplest way to run compaction on MOR table is to run the [compaction inline](configurations#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted most o [...]
+Simplest way to run compaction on MOR table is to run the [compaction inline](configurations.md#hoodiecompactinline), at the cost of spending more time ingesting; This could be particularly useful, in common cases where you have small amount of late arriving data trickling into older partitions. In such a scenario, you may want to just aggressively compact the last N partitions while waiting for enough logs to accumulate for older partitions. The net effect is that you have converted mos [...]
That said, for obvious reasons of not blocking ingesting for compaction, you may want to run it asynchronously as well. This can be done either via a separate [compaction job](https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java) that is scheduled by your workflow scheduler/notebook independently. If you are using Hudi Streamer, then you can run in [continuous mode](https://github.com/apache/hudi/blob/f5f0ef6549fedf9386352 [...]
@@ -40,7 +40,7 @@ Depending on how you write to Hudi these are the possible options currently.
### How to disable all table services in case of multiple writers?
-[hoodie.table.services.enabled](configurations#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
+[hoodie.table.services.enabled](configurations.md#hoodietableservicesenabled) is an umbrella config that can be used to turn off all table services at once without having to individually disable them. This is handy in use cases where there are multiple writers doing ingestion. While one of the main pipelines can take care of the table services, other ingestion pipelines can disable them to avoid frequent trigger of cleaning/clustering etc. This does not apply to singe writer scenarios.
### Why does Hudi retain at-least one previous commit even after setting hoodie.cleaner.commits.retained': 1 ?
@@ -50,6 +50,6 @@ Hudi runs cleaner to remove old file versions as part of writing data either in
Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
-be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback)
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback.md)
for details
diff --git a/website/versioned_docs/version-1.0.2/faq_writing_tables.md b/website/versioned_docs/version-1.0.2/faq_writing_tables.md
index 5d33a88d9dff..3c1130437e8a 100644
--- a/website/versioned_docs/version-1.0.2/faq_writing_tables.md
+++ b/website/versioned_docs/version-1.0.2/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API and us [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](write_operations.md) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API a [...]
### How is a Hudi writer job deployed?
@@ -68,21 +68,21 @@ As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github
### How do I delete records in the dataset using Hudi?
-GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data#deletes).
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data.md#deletes).
### Should I need to worry about deleting all copies of the records in case of duplicates?
-No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https://githu [...]
+No. Hudi removes all the copies of a record key when deletes are issued. Here is the long form explanation - Sometimes accidental user errors can lead to duplicates introduced into a Hudi table by either [concurrent inserts](faq_writing_tables.md#can-concurrent-inserts-cause-duplicates) or by [not deduping the input records](faq_writing_tables.md#can-single-writer-inserts-have-duplicates) for an insert operation. However, using the right index (e.g., in the default [Simple Index](https:/ [...]
### How does Hudi handle duplicate record keys in an input?
-When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
+When issuing an `upsert` operation on a table and the batch of records provided contains multiple entries for a given key, then all of them are reduced into a single final value by repeatedly calling payload class's [preCombine()](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java#L40) method . By default, we pick the record with the greatest value (determined by calling .compareTo() [...]
-For an insert or bulk_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in [...]
+For an insert or bulk_insert operation, no such pre-combining is performed. Thus, if your input contains duplicates, the table would also contain duplicates. If you don't want duplicate records either issue an **upsert** or consider specifying option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or [...]
### How can I pass hudi configurations to my spark writer job?
-Hudi configuration options covering the datasource and low level Hudi write client (which both Hudi Streamer & datasource internally call) are [here](configurations/). Invoking _--help_ on any tool such as Hudi Streamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
+Hudi configuration options covering the datasource and low level Hudi write client (which both Hudi Streamer & datasource internally call) are [here](configurations.md). Invoking _--help_ on any tool such as Hudi Streamer would print all the usage options. A lot of the options that control upsert, file sizing behavior are defined at the write client level and below is how we pass them to different options available for writing data.
* For Spark DataSource, you can use the "options" API of DataFrameWriter to pass in these configs.
@@ -108,28 +108,28 @@ hoodie.datasource.write.hive_style_partitioning: true
### Can I register my Hudi table with Apache Hive metastore?
-Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations#hoodiedatasourcehive_syncenable).
+Yes. This can be performed either via the standalone [Hive Sync tool](syncing_metastore.md#hive-sync-tool) or using options in [Hudi Streamer](https://github.com/apache/hudi/blob/d3edac4612bde2fa9deca9536801dbc48961fb95/docker/demo/sparksql-incremental.commands#L50) tool or [datasource](configurations.md#hoodiedatasourcehive_syncenable).
### What's Hudi's schema evolution story?
Hudi uses Avro as the internal canonical representation for records, primarily due to its nice [schema compatibility & evolution](https://docs.confluent.io/platform/current/schema-registry/avro.html) properties. This is a key aspect of having reliability in your ingestion or ETL pipelines. As long as the schema passed to Hudi (either explicitly in Hudi Streamer schema provider configs or implicitly by Spark Datasource's Dataset schemas) is backwards compatible (e.g no field deletes, only [...]
-Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution) for more details on Schema-on-read for Spark..
+Starting 0.11.0, Spark SQL DDL support (experimental) was added for Spark 3.1.x and Spark 3.2.1 via ALTER TABLE syntax. Please refer to the [schema evolution guide](schema_evolution.md) for more details on Schema-on-read for Spark..
### What performance/ingest latency can I expect for Hudi writing?
-The speed at which you can write into Hudi depends on the [write operation](write_operations) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when try [...]
+The speed at which you can write into Hudi depends on the [write operation](write_operations.md) and some trade-offs you make along the way like file sizing. Just like how databases incur overhead over direct/raw file I/O on disks, Hudi operations may have overhead from supporting database like features compared to reading/writing raw DFS files. That said, Hudi implements advanced techniques from database literature to keep these minimal. User is encouraged to have this perspective when [...]
| Storage Type | Type of workload | Performance | Tips |
| ---| ---| ---| --- |
-| copy on write | bulk_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism) to get right number of files. Use insert if you want this auto tuned. Configure [hoodie.bulkinsert.sort.mode](configurations#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value `NONE` offers the fastest performance and matches `spark.write.parquet()` [...]
-| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
-| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or less than 50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For example, if a partitio [...]
+| copy on write | bulk_insert | Should match vanilla spark writing + an additional sort to properly size files | properly size [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism) to get right number of files. Use insert if you want this auto tuned. Configure [hoodie.bulkinsert.sort.mode](configurations.md#hoodiebulkinsertsortmode) for better file sizes at the cost of memory. The default value `NONE` offers the fastest performance and matches `spark.write.parqu [...]
+| copy on write | insert | Similar to bulk insert, except the file sizes are auto tuned requiring input to be cached into memory and custom partitioned. | Performance would be bound by how parallel you can write the ingested data. Tune [this limit](configurations.md#hoodieinsertshuffleparallelism) up, if you see that writes are happening from only a few executors. |
+| copy on write | upsert/ de-duplicate & insert | Both of these would involve index lookup. Compared to naively using Spark (or similar framework)'s JOIN to identify the affected records, Hudi indexing is often 7-10x faster as long as you have ordered keys (discussed below) or less than 50% updates. Compared to naively overwriting entire partitions, Hudi write can be several magnitudes faster depending on how many files in a given partition is actually updated. For example, if a partitio [...]
| merge on read | bulk insert | Currently new data only goes to parquet files and thus performance here should be similar to copy on write bulk insert. This has the nice side-effect of getting data into parquet directly for query performance. [This GitHub issue](https://github.com/apache/hudi/issues/14468) will add support for logging inserts directly and this up drastically. | |
| merge on read | insert | Similar to above | |
| merge on read | upsert/ de-duplicate & insert | Indexing performance would remain the same as copy-on-write, while ingest latency for updates (costliest I/O operation in copy on write) are sent to log files and thus with asynchronous compaction provides very good ingest performance with low write amplification. | |
-Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide) for more tips on JVM and other configurations.
+Like with many typical system that manage time-series data, Hudi performs much better if your keys have a timestamp prefix or monotonically increasing/decreasing. You can almost always achieve this. Even if you have UUID keys, you can follow tricks like [this](https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/) to get keys that are ordered. See also [Tuning Guide](tuning-guide.md) for more tips on JVM and other configurations.
### What performance can I expect for Hudi reading/queries?
@@ -147,11 +147,11 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations#hoodieparquetmaxfilesize) and the [soft limit](configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on previou [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](configurations.md#hoodieparquetmaxfilesize) and the [soft limit](configurations.md#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size based on p [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
-* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations#hoodielogfilemaxsize) and a [factor](configurations#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
+* Indexes with **canIndexLogFiles = true** : Inserts of new data go directly to log files. In this case, you can configure the [maximum log size](configurations.md#hoodielogfilemaxsize) and a [factor](configurations.md#hoodielogfiletoparquetcompressionratio) that denotes reduction in size when data moves from avro to parquet files.
* Indexes with **canIndexLogFiles = false** : Inserts of new data go only to parquet files. In this case, the same configurations as above for the COPY_ON_WRITE case applies.
NOTE : In either case, small files will be auto sized only if there is no PENDING compaction or associated log file for that particular file slice. For example, for case 1: If you had a log file and a compaction C1 was scheduled to convert that log file to parquet, no more inserts can go into that log file. For case 2: If you had a parquet file and an update ended up creating an associated delta log file, no more inserts can go into that parquet file. Only after the compaction has been p [...]
@@ -173,7 +173,7 @@ hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPa
With each commit, Hudi creates a new table version in the metastore. This can be reduced by setting the option
-[hoodie.datasource.meta_sync.condition.sync](configurations#hoodiedatasourcemeta_syncconditionsync) to true.
+[hoodie.datasource.meta_sync.condition.sync](configurations.md#hoodiedatasourcemeta_syncconditionsync) to true.
This will ensure that hive sync is triggered on schema or partitions changes.
@@ -183,11 +183,11 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
+Hudi employs [optimistic concurrency control](concurrency_control.md) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in β[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/ [...]
### Can single-writer inserts have duplicates?
-By default, Hudi turns off key based de-duplication for INSERT/BULK_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in Hudi [...]
+By default, Hudi turns off key based de-duplication for INSERT/BULK_INSERT operations and thus the table could contain duplicates. If users believe, they have duplicates in inserts, they can either issue UPSERT or consider specifying the option to de-duplicate input in either datasource using [`hoodie.datasource.write.insert.drop.duplicates`](configurations.md#hoodiedatasourcewriteinsertdropduplicates) & [`hoodie.combine.before.insert`](configurations/#hoodiecombinebeforeinsert) or in Hu [...]
### Can concurrent inserts cause duplicates?
diff --git a/website/versioned_docs/version-1.0.2/file_sizing.md b/website/versioned_docs/version-1.0.2/file_sizing.md
index 62ad0f7a4320..02ec64a8fe43 100644
--- a/website/versioned_docs/version-1.0.2/file_sizing.md
+++ b/website/versioned_docs/version-1.0.2/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning.md)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-1.0.2/flink-quick-start-guide.md b/website/versioned_docs/version-1.0.2/flink-quick-start-guide.md
index 6c592fa537b2..d0071cc57190 100644
--- a/website/versioned_docs/version-1.0.2/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.2/flink-quick-start-guide.md
@@ -60,7 +60,7 @@ quick start tool for SQL users.
### Start Flink SQL client
Hudi supports packaged bundle jar for Flink, which should be loaded in the Flink SQL Client when it starts up.
-You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore#install)), or download it from the
+You can build the jar manually under path `hudi-source-dir/packaging/hudi-flink-bundle`(see [Build Flink Bundle Jar](syncing_metastore.md#install)), or download it from the
[Apache Official Repository](https://repo.maven.apache.org/maven2/org/apache/hudi/).
Now start the SQL CLI:
@@ -282,7 +282,7 @@ Refer Full Streaming Reader Example [here](https://github.com/ad1happy2go/hudi-e
>
This statement queries snapshot view of the dataset.
-Refers to [Table types and queries](concepts#table-types--queries) for more info on all table types and query types supported.
+Refers to [Table types and queries](concepts.md#table-types--queries) for more info on all table types and query types supported.
## Update Data
@@ -325,7 +325,7 @@ Refer Update Example [here](https://github.com/ad1happy2go/hudi-examples/blob/ma
</Tabs
>
-[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts)
+[Querying](#query-data) the data again will now show updated records. Each write operation generates a new [commit](concepts.md)
denoted by the timestamp.
@@ -449,19 +449,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink#cdc-ingestion), [Bulk Insert](ingestion_flink#bulk-insert), [Index Bootstrap](ingestion_flink#index-bootstrap), [Changelog Mode](ingestion_flink#changelog-mode) and [Append Mode](ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml#non-blocking-concurrency-control-experimental).
-- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](sql_queries#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
-- **Optimization**: Offline compaction is supported [Offline Compaction](compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
+- **Configuration** : For [Global Configuration](flink_tuning.md#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning.md#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk Insert](ingestion_flink.md#bulk-insert), [Index Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog Mode](ingestion_flink.md#changelog-mode) and [Append Mode](ingestion_flink.md#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml.md#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries.md#streaming-query) and [Incremental Query](sql_queries.md#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization) and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Optimization**: Offline compaction is supported [Offline Compaction](compaction.md#flink-offline-compaction).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
- - [Hudi Storage Layout](storage_layouts) - How the files are laid out on storage
- - [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+ - [Hudi Storage Layout](storage_layouts.md) - How the files are laid out on storage
+ - [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
@@ -470,5 +470,5 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Hudi tables can be queried from query engines like Hive, Spark, Flink, Presto and much more. We have put together a
[demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that show cases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it. Also, if you are looking for ways to migrate your existing data
-to Hudi, refer to [migration guide](migration_guide).
+steps [here](docker_demo.md) to get a taste for it. Also, if you are looking for ways to migrate your existing data
+to Hudi, refer to [migration guide](migration_guide.md).
diff --git a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
index 129bbc2e22b5..c922b4831d07 100644
--- a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
@@ -268,7 +268,7 @@ and then ingest it as follows.
--op BULK_INSERT
```
-In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide).
+In some cases, you may want to migrate your existing table into Hudi beforehand. Please refer to [migration guide](migration_guide.md).
### Using `hudi-utilities-slim-bundle` bundle jar
@@ -303,7 +303,7 @@ A Hudi Streamer job can then be triggered as follows:
--op BULK_INSERT
```
-Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control) section
+Read more in depth about concurrency control in the [concurrency control concepts](concurrency_control.md) section
### Checkpointing
@@ -314,7 +314,7 @@ Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single [...]
+- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a sing [...]
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
diff --git a/website/versioned_docs/version-1.0.2/hudi_stack.md b/website/versioned_docs/version-1.0.2/hudi_stack.md
index 214a1ba2f7aa..bd5d3e5cf152 100644
--- a/website/versioned_docs/version-1.0.2/hudi_stack.md
+++ b/website/versioned_docs/version-1.0.2/hudi_stack.md
@@ -49,19 +49,19 @@ bring any compute engine for specific workloads.
Drawing an analogy to file formats, a table format simply concerns with how files are distributed with the table, partitioning schemes, schema and metadata tracking changes. Hudi organizes files within a table or partition into
File Groups. Updates are captured in log files tied to these File Groups, ensuring efficient merges. There are three major components related to Hudiβs table format.
-- **Timeline** : Hudi's [timeline](timeline), stored in the `/.hoodie/timeline` folder, is a crucial event log recording all table actions in an ordered manner,
+- **Timeline** : Hudi's [timeline](timeline.md), stored in the `/.hoodie/timeline` folder, is a crucial event log recording all table actions in an ordered manner,
with events kept for a specified period. Hudi uniquely designs each File Group as a self-contained log, enabling record state reconstruction through delta logs, even after archival of historical actions. This approach effectively limits metadata size based on table activity frequency, essential for managing tables with frequent updates.
- **File Group and File Slice** : Within each partition the data is physically stored as base and Log Files and organized into logical concepts as [File groups](https://hudi.apache.org/tech-specs-1point0/#storage-layout) and
File Slices. File groups contain multiple versions of File Slices and are split into multiple File Slices. A File Slice comprises the Base and Log File. Each File Slice within
the file-group is uniquely identified by the write that created its base file or the first log file, which helps order the File Slices.
-- **Metadata Table** : Implemented as another merge-on-read Hudi table, the [metadata table](metadata) efficiently handles quick updates with low write amplification.
+- **Metadata Table** : Implemented as another merge-on-read Hudi table, the [metadata table](metadata.md) efficiently handles quick updates with low write amplification.
It leverages a [SSTable](https://cassandra.apache.org/doc/stable/cassandra/architecture/storage-engine.html#sstables) based file format for quick, indexed key lookups,
storing vital information like file paths, column statistics and schema. This approach streamlines operations by reducing the necessity for expensive cloud file listings.
Hudiβs approach of recording updates into Log Files is more efficient and involves low merge overhead than systems like Hive ACID, where merging all delta records against
-all Base Files is required. Read more about the various table types in Hudi [here](table_types).
+all Base Files is required. Read more about the various table types in Hudi [here](table_types.md).
## Storage Engine
@@ -74,8 +74,8 @@ Cassandra and Clickhouse.
<p align = "center">Figure: Indexes in Hudi</p>
-[Indexes](indexes) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. The [metadata table](metadata/#metadata-table-indices) acts
-as an additional [indexing system](metadata#supporting-multi-modal-index-in-hudi) and brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata
+[Indexes](indexes.md) in Hudi enhance query planning, minimizing I/O, speeding up response times and providing faster writes with low merge costs. The [metadata table](metadata/#metadata-table-indices) acts
+as an additional [indexing system](metadata.md#supporting-multi-modal-index-in-hudi) and brings the benefits of indexes generally to both the readers and writers. Compute engines can leverage various indexes in the metadata
table, like file listings, column statistics, bloom filters, record-level indexes, and [expression indexes](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) to quickly generate optimized query plans and improve read
performance. In addition to the metadata table indexes, Hudi supports simple join based indexing, bloom filters stored in base file footers, external key-value stores like HBase,
and optimized storage techniques like bucketing , to efficiently locate File Groups containing specific record keys. Hudi also provides reader indexes such as [expression](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) and
@@ -91,12 +91,12 @@ running them in inline, semi-asynchronous or full-asynchronous modes. Furthermor
asynchronously sharing the underlying executors intelligently with writers. Letβs take a look at these services.
#### Clustering
-The [clustering](clustering) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into
+The [clustering](clustering.md) service, akin to features in cloud data warehouses, allows users to group frequently queried records using sort keys or merge smaller Base Files into
larger ones for optimal file size management. It's fully integrated with other timeline actions like cleaning and compaction, enabling smart optimizations such as avoiding
compaction for File Groups undergoing clustering, thereby saving on I/O.
#### Compaction
-Hudi's [compaction](compaction) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows
+Hudi's [compaction](compaction.md) service, featuring strategies like date partitioning and I/O bounding, merges Base Files with delta logs to create updated Base Files. It allows
concurrent writes to the same File Froup, enabled by Hudi's file grouping and flexible log merging. This facilitates non-blocking execution of deletes even during concurrent
record updates.
@@ -107,11 +107,11 @@ while also allowing sufficient time for long running batch jobs (e.g Hive ETLs)
#### Indexing
Hudi's scalable metadata table contains auxiliary data about the table. This subsystem encompasses various indices, including files, column_stats, and bloom_filters,
facilitating efficient record location and data skipping. Balancing write throughput with index updates presents a fundamental challenge, as traditional indexing methods,
-like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchronous [metadata indexing](metadata_indexing),
+like locking out writes during indexing, are impractical for large tables due to lengthy processing times. Hudi addresses this with its innovative asynchronous [metadata indexing](metadata_indexing.md),
enabling the creation of various indices without impeding writes. This approach not only improves write latency but also minimizes resource waste by reducing contention between writing and indexing activities.
### Concurrency Control
-[Concurrency control](concurrency_control) defines how different writers/readers/table services coordinate access to the table. Hudi uses monotonically increasing time to sequence and order various
+[Concurrency control](concurrency_control.md) defines how different writers/readers/table services coordinate access to the table. Hudi uses monotonically increasing time to sequence and order various
changes to table state. Much like databases, Hudi take an approach of clearly differentiating between writers (responsible for upserts/deletes), table services
(focusing on storage optimization and bookkeeping), and readers (for query execution). Hudi provides snapshot isolation, offering a consistent view of the table across
these different operations. It employs lock-free, non-blocking MVCC for concurrency between writers and table-services, as well as between different table services, and
@@ -154,12 +154,12 @@ integration with engines written in C/C++.
<p align = "center">Figure: Various platform services in Hudi</p>
Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers.
-Services, like [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
+Services, like [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (including Confluent), and
deduplication of data. Hudi Streamer also offers features for backfills, one-off runs, and continuous mode operation with Spark/Flink streaming writers. Additionally,
-Hudi provides tools for [snapshotting](snapshot_exporter) and incrementally [exporting](snapshot_exporter#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback) for analytics or
+Hudi provides tools for [snapshotting](snapshot_exporter.md) and incrementally [exporting](snapshot_exporter.md#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback.md) for analytics or
workflow management, enhancing the deployment of production-grade incremental pipelines. Apart from these services, Hudi also provides broad support for different
-catalogs such as [Hive Metastore](syncing_metastore), [AWS Glue](syncing_aws_glue_data_catalog/), [Google BigQuery](gcp_bigquery), [DataHub](syncing_datahub), etc. that allows syncing of Hudi tables to be queried by
+catalogs such as [Hive Metastore](syncing_metastore.md), [AWS Glue](syncing_aws_glue_data_catalog.md), [Google BigQuery](gcp_bigquery.md), [DataHub](syncing_datahub.md), etc. that allows syncing of Hudi tables to be queried by
interactive engines such as Trino and Presto.
### Metaserver*
diff --git a/website/versioned_docs/version-1.0.2/indexes.md b/website/versioned_docs/version-1.0.2/indexes.md
index 43a6bab9c631..aaec4bece409 100644
--- a/website/versioned_docs/version-1.0.2/indexes.md
+++ b/website/versioned_docs/version-1.0.2/indexes.md
@@ -19,8 +19,8 @@ Only clustering or cross-partition updates that are implemented as deletes + ins
file group at any completed instant on the timeline.
## Need for indexing
-For [Copy-On-Write tables](table_types#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](table_types#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
+For [Copy-On-Write tables](table_types.md#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
+For [Merge-On-Read tables](table_types.md#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
only against updates for records that are part of that base file.
@@ -28,7 +28,7 @@ only against updates for records that are part of that base file.
In contrast,
- Designs without an indexing component (e.g: [Apache Hive/Apache Iceberg](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)) end up having to merge all the base files against all incoming updates/delete records
- (10-100x more [read amplification](table_types#comparison)).
+ (10-100x more [read amplification](table_types.md#comparison)).
- Designs that implement heavily write-optimized OLTP data structures like LSM trees do not require an indexing component. But they perform poorly scan heavy workloads
against cloud storage making them unsuitable for serving analytical queries.
@@ -42,8 +42,8 @@ implemented by enhancing the metadata table with the flexibility to extend to ne
along with an [asynchronous index](https://hudi.apache.org/docs/metadata_indexing/#setup-async-indexing) building
Hudi supports a multi-modal index by augmenting the metadata table with the capability to incorporate new types of indexes, complemented by an
-asynchronous mechanism for [index construction](metadata_indexing). This enhancement supports a range of indexes within
-the [metadata table](metadata#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
+asynchronous mechanism for [index construction](metadata_indexing.md). This enhancement supports a range of indexes within
+the [metadata table](metadata.md#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
<p align = "center">Figure: Indexes in Hudi</p>
@@ -90,7 +90,7 @@ Following are configurations that control enabling record index building and mai
An [expression index](https://github.com/apache/hudi/blob/3789840be3d041cbcfc6b24786740210e4e6d6ac/rfc/rfc-63/rfc-63.md) is an index on a function of a column. If a query has a predicate on a function of a column, the expression index can
be used to speed up the query. Expression index is stored in *expr_index_* prefixed partitions (one for each
expression index) under metadata table. Expression index can be created using SQL syntax. Please checkout SQL DDL
-docs [here](sql_ddl#create-expression-index) for more details.
+docs [here](sql_ddl.md#create-expression-index) for more details.
Following are configurations that control enabling expression index building and maintenance on the writer.
diff --git a/website/versioned_docs/version-1.0.2/ingestion_flink.md b/website/versioned_docs/version-1.0.2/ingestion_flink.md
index 8176ff1e5ac6..8ec300910490 100644
--- a/website/versioned_docs/version-1.0.2/ingestion_flink.md
+++ b/website/versioned_docs/version-1.0.2/ingestion_flink.md
@@ -37,7 +37,7 @@ frequent `file handle` switching.
:::note
The parallelism of `bulk_insert` is specified by `write.tasks`. The parallelism will affect the number of small files.
In theory, the parallelism of `bulk_insert` is the number of `bucket`s (In particular, when each bucket writes to maximum file size, it
-will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks).
+will rollover to the new file handle. Finally, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks).
:::
#### Options
@@ -45,7 +45,7 @@ will rollover to the new file handle. Finally, `the number of files` >= [`write.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.operation` | `true` | `upsert` | Setting as `bulk_insert` to open this function |
-| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations#writebucket_assigntasks) |
+| `write.tasks` | `false` | `4` | The parallelism of `bulk_insert`, `the number of files` >= [`write.bucket_assign.tasks`](configurations.md#writebucket_assigntasks) |
| `write.bulk_insert.shuffle_input` | `false` | `true` | Whether to shuffle data according to the input field before writing. Enabling this option will reduce the number of small files, but there may be a risk of data skew |
| `write.bulk_insert.sort_input` | `false` | `true` | Whether to sort data according to the input field before writing. Enabling this option will reduce the number of small files when a write task writes multiple partitions |
| `write.sort.memory` | `false` | `128` | Available managed memory of sort operator. default `128` MB |
diff --git a/website/versioned_docs/version-1.0.2/metadata.md b/website/versioned_docs/version-1.0.2/metadata.md
index a9df1ffc82d9..d50d3999e163 100644
--- a/website/versioned_docs/version-1.0.2/metadata.md
+++ b/website/versioned_docs/version-1.0.2/metadata.md
@@ -62,12 +62,12 @@ Following are the different types of metadata currently supported.
even without examining column statistics at the file level. The partition stats index is stored in *partition_stats* partition in the metadata table.
Partition stats index can be enabled using the following configs (note it is required to specify the columns for which stats should be aggregated).
-To try out these features, refer to the [SQL guide](sql_ddl#create-partition-stats-index).
+To try out these features, refer to the [SQL guide](sql_ddl.md#create-partition-stats-index).
## Metadata Tracking on Writers
Following are based basic configs that are needed to enable metadata tracking. For advanced configs please refer
-[here](configurations#Metadata-Configs).
+[here](configurations.md#Metadata-Configs).
| Config Name | Default | Description |
|----------------------------------------------|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -118,10 +118,10 @@ corresponding configs across Spark and Flink readers.
To ensure that metadata table stays up to date and table metadata is tracked safely across concurrent write and
table operations, there are some additional considerations. If async table services are enabled for the table (i.e. running a separate compaction (`HoodieCompactor`) or
clustering (`HoodieClusteringJob`) job), even with just a single writer, lock providers
-must be configured. Please refer to [concurrency control](concurrency_control) for more details.
+must be configured. Please refer to [concurrency control](concurrency_control.md) for more details.
Before enabling metadata table for the first time, all writers on the same table must and table services must be stopped.
-If your current deployment model is [multi-writer](concurrency_control#full-on-multi-writer--async-table-services) along with a lock
+If your current deployment model is [multi-writer](concurrency_control.md#full-on-multi-writer--async-table-services) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only a subset of writers or table services is unsafe and can lead to loss of data. So, please ensure you enable
diff --git a/website/versioned_docs/version-1.0.2/metadata_indexing.md b/website/versioned_docs/version-1.0.2/metadata_indexing.md
index 29959625cb35..a34df6f476dc 100644
--- a/website/versioned_docs/version-1.0.2/metadata_indexing.md
+++ b/website/versioned_docs/version-1.0.2/metadata_indexing.md
@@ -5,7 +5,7 @@ toc: true
last_modified_at:
---
-Hudi maintains a scalable [metadata](metadata) that has some auxiliary data about the table.
+Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data about the table.
The [pluggable indexing subsystem](https://www.onehouse.ai/blog/introducing-multi-modal-index-for-the-lakehouse-in-apache-hudi)
of Hudi depends on the metadata table. Different types of index, from `files` index for locating records efficiently
to `column_stats` index for data skipping, are part of the metadata table. A fundamental tradeoff in any data system
@@ -35,7 +35,7 @@ For more information on these indexes please refer [metadata section](metadata/#
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -54,7 +54,7 @@ CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(
CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
```
-For more information on index creation using SQL refer [SQL DDL](sql_ddl#create-index)
+For more information on index creation using SQL refer [SQL DDL](sql_ddl.md#create-index)
## Index Creation Using Datasource
@@ -129,7 +129,7 @@ In the example we will have continuous writing using Hudi Streamer and also crea
in example is done using HoodieIndexer so that schedule and execute phases are clearly visible for indexing. The asynchronous
configurations can be used with Datasource and SQL based configs to create index as well.
-First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) which will continuously write data
+First, we will generate a continuous workload. In the below example, we are going to start a [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) which will continuously write data
from raw parquet to Hudi table. We used the widely available [NY Taxi dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/), whose setup details are as below:
<details>
<summary>Ingestion write config</summary>
@@ -183,7 +183,7 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](indexes#multi-modal-indexing) along with [configs](metadata#metadata-tracking-on-writers)
+metadata table can be explored [here](indexes.md#multi-modal-indexing) along with [configs](metadata.md#metadata-tracking-on-writers)
to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
diff --git a/website/versioned_docs/version-1.0.2/metrics.md b/website/versioned_docs/version-1.0.2/metrics.md
index c6f2833f5ada..c389689716be 100644
--- a/website/versioned_docs/version-1.0.2/metrics.md
+++ b/website/versioned_docs/version-1.0.2/metrics.md
@@ -6,7 +6,7 @@ toc: true
last_modified_at: 2020-06-20T15:59:57-04:00
---
-In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations#METRICS).
+In this section, we will introduce the `MetricsReporter` and `HoodieMetrics` in Hudi. You can view the metrics-related configurations [here](configurations.md#METRICS).
## MetricsReporter
@@ -17,7 +17,7 @@ MetricsReporter provides APIs for reporting `HoodieMetrics` to user-specified ba
JmxMetricsReporter is an implementation of JMX reporter, which used to report JMX metrics.
#### Configurations
-The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Jmx).
+The following is an example of `JmxMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Jmx).
```properties
hoodie.metrics.on=true
@@ -37,7 +37,7 @@ As configured above, JmxMetricsReporter will started JMX server on port 4001. We
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server, and send `HoodieMetrics` to it.
#### Configurations
-The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations#Metrics-Configurations-for-Graphite).
+The following is an example of `MetricsGraphiteReporter`. More detaile configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Graphite).
```properties
hoodie.metrics.on=true
@@ -58,7 +58,7 @@ DatadogMetricsReporter is an implementation of Datadog reporter.
A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
#### Configurations
-The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations-for-Datadog-reporter).
+The following is an example of `DatadogMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations-for-Datadog-reporter).
```properties
hoodie.metrics.on=true
@@ -138,7 +138,7 @@ tuned are in the `HoodieMetricsCloudWatchConfig` class.
Allows users to define a custom metrics reporter.
#### Configurations
-The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations#Metrics-Configurations).
+The following is an example of `UserDefinedMetricsReporter`. More detailed configurations can be referenced [here](configurations.md#Metrics-Configurations).
```properties
hoodie.metrics.on=true
diff --git a/website/versioned_docs/version-1.0.2/migration_guide.md b/website/versioned_docs/version-1.0.2/migration_guide.md
index 99bafee92a75..64d87622183b 100644
--- a/website/versioned_docs/version-1.0.2/migration_guide.md
+++ b/website/versioned_docs/version-1.0.2/migration_guide.md
@@ -40,7 +40,7 @@ There are a few options when choosing this approach.
#### Using Hudi Streamer
-Use the [Hudi Streamer](hoodie_streaming_ingestion#hudi-streamer) tool. HoodieStreamer supports bootstrap with
+Use the [Hudi Streamer](hoodie_streaming_ingestion.md#hudi-streamer) tool. HoodieStreamer supports bootstrap with
--run-bootstrap command line option. There are two types of bootstrap, METADATA_ONLY and FULL_RECORD. METADATA_ONLY will
generate just skeleton base files with keys/footers, avoiding full cost of rewriting the dataset. FULL_RECORD will
perform a full copy/rewrite of the data as a Hudi table. Additionally, once can choose selective partitions using regex
@@ -89,7 +89,7 @@ Refer to [Bootstrap procedure](https://hudi.apache.org/docs/next/procedures#boot
#### Using Hudi CLI
Write your own custom logic of how to load an existing table into a Hudi managed one. Please read about the RDD API
-[here](quick-start-guide). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
+[here](quick-start-guide.md). Using the bootstrap run CLI. Once hudi has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hudi-cli && ./hudi-cli.sh`.
```java
diff --git a/website/versioned_docs/version-1.0.2/overview.mdx b/website/versioned_docs/version-1.0.2/overview.mdx
index 8f4e35c7fde7..fe320b347e16 100644
--- a/website/versioned_docs/version-1.0.2/overview.mdx
+++ b/website/versioned_docs/version-1.0.2/overview.mdx
@@ -17,14 +17,14 @@ Apache Hudi (pronounced "hoodie") pioneered the concept of "[transactional data
the data lakehouse architecture. Today, Hudi has grown into an [open data lakehouse platform](/blog/2021/07/21/streaming-data-lake-platform), with a open table format purpose-built for high performance writes on
incremental data pipelines and fast query performance due to comprehensive table optimizations.
-Hudi brings core database functionality directly to a data lake - [tables](sql_ddl),
-[transactions](timeline), [efficient upserts/deletes](write_operations), [advanced indexes](indexes),
-[ingestion services](hoodie_streaming_ingestion), data [clustering](clustering)/[compaction](compaction) optimizations,
-and [concurrency control](concurrency_control) all while keeping your data in open file formats. Not only is Apache Hudi great for streaming workloads,
-but it also allows you to create efficient incremental batch pipelines. Apache Hudi can easily be used on any [cloud storage platform](cloud).
+Hudi brings core database functionality directly to a data lake - [tables](sql_ddl.md),
+[transactions](timeline.md), [efficient upserts/deletes](write_operations.md), [advanced indexes](indexes.md),
+[ingestion services](hoodie_streaming_ingestion.md), data [clustering](clustering.md)/[compaction](compaction.md) optimizations,
+and [concurrency control](concurrency_control.md) all while keeping your data in open file formats. Not only is Apache Hudi great for streaming workloads,
+but it also allows you to create efficient incremental batch pipelines. Apache Hudi can easily be used on any [cloud storage platform](cloud.md).
Hudiβs advanced performance optimizations, make analytical queries/pipelines faster with any of the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.
-Read the docs for more [use case descriptions](use_cases) and check out [who's using Hudi](/powered-by), to see how some of the
+Read the docs for more [use case descriptions](use_cases.md) and check out [who's using Hudi](/powered-by), to see how some of the
largest data lakes in the world including [Uber](https://www.uber.com/en-IN/blog/uber-big-data-platform/), [Amazon](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/),
[ByteDance](http://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance),
[Robinhood](https://s.apache.org/hudi-robinhood-talk) and more are transforming their production data lakes with Hudi.
@@ -35,10 +35,10 @@ expands the use of Apache Hudi for a diverse range of use cases in the non-JVM e
## Core Concepts to Learn
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
-- [Hudi Timeline](timeline) β How Hudi manages transactions and other table services
-- [Hudi File Layout](storage_layouts) - How the files are laid out on storage
-- [Hudi Table Types](table_types) β `COPY_ON_WRITE` and `MERGE_ON_READ`
-- [Hudi Query Types](table_types#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
+- [Hudi Timeline](timeline.md) β How Hudi manages transactions and other table services
+- [Hudi File Layout](storage_layouts.md) - How the files are laid out on storage
+- [Hudi Table Types](table_types.md) β `COPY_ON_WRITE` and `MERGE_ON_READ`
+- [Hudi Query Types](table_types.md#query-types) β Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Design & Concepts" section of the docs.
@@ -48,11 +48,11 @@ Take a look at recent [blog posts](/blog) that go in depth on certain topics or
Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:
-- [Spark Quick Start Guide](quick-start-guide) β if you primarily use Apache Spark
-- [Flink Quick Start Guide](flink-quick-start-guide) β if you primarily use Apache Flink
-- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide) - if you primarily use Python or Rust
+- [Spark Quick Start Guide](quick-start-guide.md) β if you primarily use Apache Spark
+- [Flink Quick Start Guide](flink-quick-start-guide.md) β if you primarily use Apache Flink
+- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide.md) - if you primarily use Python or Rust
-If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo)
+If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo.md)
## Connect With The Community
Apache Hudi is community-focused and community-led and welcomes new-comers with open arms. Leverage the following
diff --git a/website/versioned_docs/version-1.0.2/performance.md b/website/versioned_docs/version-1.0.2/performance.md
index 95e505392baa..0eabc8deab23 100644
--- a/website/versioned_docs/version-1.0.2/performance.md
+++ b/website/versioned_docs/version-1.0.2/performance.md
@@ -16,12 +16,12 @@ Here are some ways to efficiently manage the storage of your Hudi tables.
- The [small file handling feature](configurations/#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](configurations#hoodieparquetcompressionratio),
+- Cleaner can be [configured](configurations.md#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](configurations.md#hoodieparquetmaxfilesize), [log files](configurations.md#hoodielogfilemaxsize) & expected [compression ratio](configurations.md#hoodieparquetcompressionratio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
-- Intelligently tuning the [bulk insert parallelism](configurations#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
+- Intelligently tuning the [bulk insert parallelism](configurations.md#hoodiebulkinsertshuffleparallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be changed without re-clustering the table. Writes will simply expand given file groups with new updates/inserts as explained before.
-- For workloads with heavy updates, the [merge-on-read table](concepts#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
+- For workloads with heavy updates, the [merge-on-read table](concepts.md#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction.
## Performance Optimizations
@@ -119,11 +119,11 @@ For very large tables (1Tb+, 10s of 1000s of files), Data skipping could
To unlock the power of Data Skipping you will need to
-1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
+1. Enable Metadata Table along with Column Stats Index on the _write path_ (See [Metadata Indexing](metadata_indexing.md)), using `hoodie.metadata.enable=true` (to enable Metadata Table on the write path, enabled by default)
2. Enable Data Skipping in your queries, using `hoodie.metadata.index.column.stats.enable=true` (to enable Column Stats Index being populated on the write path, disabled by default)
:::note
-If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
+If you're planning on enabling Column Stats Index for already existing table, please check out the [Metadata Indexing](metadata_indexing.md) guide on how to build Metadata Table Indices (such as Column Stats Index) for existing tables.
:::
To enable Data Skipping in your queries make sure to set following properties to `true` (on the read path):
diff --git a/website/versioned_docs/version-1.0.2/precommit_validator.md b/website/versioned_docs/version-1.0.2/precommit_validator.md
index f810b36f9b53..fe0a0dd77605 100644
--- a/website/versioned_docs/version-1.0.2/precommit_validator.md
+++ b/website/versioned_docs/version-1.0.2/precommit_validator.md
@@ -8,7 +8,7 @@ Data quality refers to the overall accuracy, completeness, consistency, and vali
Hudi offers **Pre-Commit Validators** that allow you to ensure that your data meets certain data quality expectations as you are writing with Hudi Streamer or Spark Datasource writers.
:::note
-Pre-commit validators are skipped when using the [BULK_INSERT](write_operations#bulk_insert) write operation type.
+Pre-commit validators are skipped when using the [BULK_INSERT](write_operations.md#bulk_insert) write operation type.
:::
Multiple class names can be separated by `,` delimiter.
@@ -105,7 +105,7 @@ void validateRecordsBeforeAndAfter(Dataset<Row> before,
```
## Additional Monitoring with Notifications
-Hudi offers a [commit notification service](platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
+Hudi offers a [commit notification service](platform_services_post_commit_callback.md) that can be configured to trigger notifications about write commits.
The commit notification service can be combined with pre-commit validators to send a notification when a commit fails a validation. This is possible by passing details about the validation as a custom value to the HTTP endpoint.
diff --git a/website/versioned_docs/version-1.0.2/procedures.md b/website/versioned_docs/version-1.0.2/procedures.md
index 6dae3a349b81..b7eb6e614352 100644
--- a/website/versioned_docs/version-1.0.2/procedures.md
+++ b/website/versioned_docs/version-1.0.2/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations.md#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations.md#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations.md#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations.md#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations.md#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1535,11 +1535,11 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
-| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
-| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [clean_policy](configurations.md#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum qu [...]
+| [retain_commits](configurations.md#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations.md#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of [...]
+| [file_versions_retained](configurations.md#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations.md#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
@@ -1635,12 +1635,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations.md#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations.md#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations.md#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations.md#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations.md#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations.md#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1850,18 +1850,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations.md#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations.md#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations.md#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
+| [parallelism](configurations.md#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-1.0.2/python-rust-quick-start-guide.md b/website/versioned_docs/version-1.0.2/python-rust-quick-start-guide.md
index 3cabe6f1726e..fcd31e544200 100644
--- a/website/versioned_docs/version-1.0.2/python-rust-quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.2/python-rust-quick-start-guide.md
@@ -22,7 +22,7 @@ cargo add hudi
> [!NOTE]
> These examples expect a Hudi table exists at `/tmp/trips_table`, created using
-> the [quick start guide](quick-start-guide).
+> the [quick start guide](quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-1.0.2/querying_data.md b/website/versioned_docs/version-1.0.2/querying_data.md
index 61f3914fd761..2bbc335724bf 100644
--- a/website/versioned_docs/version-1.0.2/querying_data.md
+++ b/website/versioned_docs/version-1.0.2/querying_data.md
@@ -7,10 +7,10 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl.md), [SQL DML](sql_dml.md), [SQL Queries](sql_queries.md) and [Procedures](procedures.md) for the latest documentation.
:::
-Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts#query-types).
+Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](concepts.md#query-types).
Once the table is synced to the Hive metastore, it provides external Hive tables backed by Hudi's custom inputformats. Once the proper hudi
bundle has been installed, the table can be queried by popular query engines like Hive, Spark SQL, Flink, Trino and PrestoDB.
@@ -19,7 +19,7 @@ In sections, below we will discuss specific setup to access different query type
## Spark Datasource
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
-See the [Spark Quick Start](quick-start-guide) for more examples of Spark datasource reading queries.
+See the [Spark Quick Start](quick-start-guide.md) for more examples of Spark datasource reading queries.
**Setup**
@@ -56,8 +56,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Incremental Queries](quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](configurations#SPARK_DATASOURCE) section, to view all datasource options.
+For examples, refer to [Incremental Queries](quick-start-guide.md#incremental-query) in the Spark quickstart.
+Please refer to [configurations](configurations.md#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-1.0.2/quick-start-guide.md b/website/versioned_docs/version-1.0.2/quick-start-guide.md
index f164db5bed8c..a853758ce573 100644
--- a/website/versioned_docs/version-1.0.2/quick-start-guide.md
+++ b/website/versioned_docs/version-1.0.2/quick-start-guide.md
@@ -214,7 +214,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl.md) reference guide.
</TabItem>
@@ -258,7 +258,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -291,7 +291,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations.md) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -300,7 +300,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml.md#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -412,7 +412,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml.md#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -442,7 +442,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts.md).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -546,7 +546,7 @@ Notice, instead of `UPDATE SET *`, we are updating only the `fare` column.
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data.md#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -626,7 +626,7 @@ Hudi supports indexing on columns to speed up queries. Indexes can be created on
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -1093,7 +1093,7 @@ Note that CDC queries are currently only supported on Copy-on-Write tables.
The examples thus far have showcased one of the two table types, that Hudi supports - Copy-on-Write (COW) tables. Hudi also supports
a more advanced write-optimized table type called Merge-on-Read (MOR) tables, that can balance read and write performance in a more
-flexible manner. See [table types](table_types) for more details.
+flexible manner. See [table types](table_types.md) for more details.
Any of these examples can be run on a Merge-on-Read table by simply changing the table type to MOR, while creating the table, as below.
@@ -1219,7 +1219,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations.md).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1231,8 +1231,8 @@ Hudi also uses this mechanism to support out-of-order data arrival into a table,
For e.g. using a _created_at_ timestamp field as the precombine field will prevent older versions of a record from overwriting newer ones or being exposed to queries, even
if they are written at a later commit time to the table. This is one of the key features, that makes Hudi, best suited for dealing with streaming data.
-To enable different merge semantics, Hudi supports [merge modes](record_merger). Commit time and event time based merge modes are supported out of the box.
-Users can also define their own custom merge strategies, see [here](sql_ddl#create-table-with-record-merge-mode).
+To enable different merge semantics, Hudi supports [merge modes](record_merger.md). Commit time and event time based merge modes are supported out of the box.
+Users can also define their own custom merge strategies, see [here](sql_ddl.md#create-table-with-record-merge-mode).
<Tabs
groupId="programming-language"
@@ -1295,29 +1295,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide.md).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
-For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl.md) and [Spark SQL DML](sql_dml.md) reference guides.
+For alter table commands, check out [this](sql_ddl.md#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures.md) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion.md#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes#spark-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes.md#spark-streaming) for more.
-Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data) and [streaming writes](writing_tables_streaming_writes).
+Check out more information on [modeling data in Hudi](faq_general.md#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](writing_data.md) and [streaming writes](writing_tables_streaming_writes.md).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](docker_demo) to get a taste for it.
+steps [here](docker_demo.md) to get a taste for it.
diff --git a/website/versioned_docs/version-1.0.2/record_merger.md b/website/versioned_docs/version-1.0.2/record_merger.md
index 8a4f0419c553..93a149a8a991 100644
--- a/website/versioned_docs/version-1.0.2/record_merger.md
+++ b/website/versioned_docs/version-1.0.2/record_merger.md
@@ -6,7 +6,7 @@ toc_min_heading_level: 2
toc_max_heading_level: 4
---
-Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline#ordering-of-actions) section.
+Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline.md#ordering-of-actions) section.
To provide users full-fledged support for stream processing, Hudi goes all the way making the storage engine and the underlying storage format
understand how to merge changes to the same record key, that may arrive even in different order at different times. With the rise of mobile applications
and IoT, these scenarios have become the normal than an exception. For e.g. a social networking application uploading user events several hours after they happened,
@@ -54,7 +54,7 @@ With event time ordering, the merging picks the record with the highest value on
In the example above, two microservices product change records about orders at different times, that can arrive out-of-order. As color coded,
this can lead to application-level inconsistent states in the table if simply merged in commit time order like a cancelled order being re-created or
a paid order moved back to just created state expecting payment again. Event time ordering helps by ignoring older state changes that arrive late and
-avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control#non-blocking-concurrency-control-mode),
+avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control.md#non-blocking-concurrency-control-mode),
this provides a very powerful way for processing such data streams efficiently and correctly.
### CUSTOM
@@ -175,7 +175,7 @@ hoodie.datasource.write.payload.class=org.apache.hudi.common.model.EventTimeAvro
```
This is the default record payload for Flink based writing. Some use cases require merging records by event time and
thus event time plays the role of an ordering field. This payload is particularly useful in the case of late-arriving data.
-For such use cases, users need to set the [payload event time field](configurations#RECORD_PAYLOAD) configuration.
+For such use cases, users need to set the [payload event time field](configurations.md#RECORD_PAYLOAD) configuration.
#### OverwriteNonDefaultsWithLatestAvroPayload
```scala
@@ -249,7 +249,7 @@ Payload class can be specified using the below configs. For more advanced config
There are also quite a few other implementations. Developers may be interested in looking at the hierarchy of `HoodieRecordPayload` interface. For
example, [`MySqlDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/MySqlDebeziumAvroPayload.java) and [`PostgresDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/PostgresDebeziumAvroPayload.java) provides support for seamlessly applying changes
captured via Debezium for MySQL and PostgresDB. [`AWSDmsAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/AWSDmsAvroPayload.java) provides support for applying changes captured via Amazon Database Migration Service onto S3.
-For full configurations, go [here](configurations#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
+For full configurations, go [here](configurations.md#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
## Related Resources
diff --git a/website/versioned_docs/version-1.0.2/s3_hoodie.md b/website/versioned_docs/version-1.0.2/s3_hoodie.md
index 88db239cf29b..ca267c1264f8 100644
--- a/website/versioned_docs/version-1.0.2/s3_hoodie.md
+++ b/website/versioned_docs/version-1.0.2/s3_hoodie.md
@@ -88,7 +88,7 @@ AWS glue data libraries are needed if AWS glue data is used
## AWS S3 Versioned Bucket
-With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](cleaning) the number of Delete Markers increases over time.
+With versioned buckets any object deleted creates a [Delete Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html), as Hudi cleans up files using [Cleaner utility](cleaning.md) the number of Delete Markers increases over time.
It is important to configure the [Lifecycle Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) correctly
to clean up these delete markers as the List operation can choke if the number of delete markers reaches 1000.
We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
diff --git a/website/versioned_docs/version-1.0.2/sql_ddl.md b/website/versioned_docs/version-1.0.2/sql_ddl.md
index d1a08846ecc0..565a62f480ef 100644
--- a/website/versioned_docs/version-1.0.2/sql_ddl.md
+++ b/website/versioned_docs/version-1.0.2/sql_ddl.md
@@ -75,7 +75,7 @@ should be specified as `PARTITIONED BY (dt, hh)`.
### Create table with record keys and ordering fields
-As discussed [here](quick-start-guide#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
+As discussed [here](quick-start-guide.md#keys), tables track each record in the table using a record key. Hudi auto-generated a highly compressed
key for each new record in the examples so far. If you want to use an existing field as the key, you can set the `primaryKey` option.
Typically, this is also accompanied by configuring a `preCombineField` option to deal with out-of-order data and potential
duplicate records with the same key in the incoming writes.
@@ -105,7 +105,7 @@ TBLPROPERTIES (
### Create table with merge modes {#create-table-with-record-merge-mode}
-Hudi supports different [record merge modes](record_merger) to handle merge of incoming records with existing
+Hudi supports different [record merge modes](record_merger.md) to handle merge of incoming records with existing
records. To create a table with specific record merge mode, you can set `recordMergeMode` option.
```sql
@@ -127,7 +127,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode/';
With `EVENT_TIME_ORDERING`, the record with the larger event time (`precombineField`) overwrites the record with the
smaller event time on the same key, regardless of transaction's commit time. Users can set `CUSTOM` mode to provide their own
merge logic. With `CUSTOM` merge mode, you can provide a custom class that implements the merge logic. The interfaces
-to implement is explained in detail [here](record_merger#custom).
+to implement is explained in detail [here](record_merger.md#custom).
```sql
CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
@@ -147,7 +147,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode_custom/';
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion.md#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -236,7 +236,7 @@ AS SELECT * FROM parquet_table;
### Create Index
Hudi supports creating and dropping different types of indexes on a table. For more information on different
-type of indexes please refer [multi-modal indexing](indexes#multi-modal-indexing). Secondary
+type of indexes please refer [multi-modal indexing](indexes.md#multi-modal-indexing). Secondary
index, expression index and record indexes can be created using SQL create index command.
```sql
@@ -270,7 +270,7 @@ Both index and column on which the index is created can be qualified with some o
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
3. Secondary index is not supported for [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types).
:::
@@ -344,7 +344,7 @@ CREATE INDEX ts_hour ON hudi_table
:::note
1. Expression index can only be created for Spark engine using SQL. It is not supported yet with Spark DataSource API.
2. Expression index is not yet supported for [complex types](https://avro.apache.org/docs/1.11.1/specification/#complex-types).
-3. Expression index is supported for unary and certain binary expressions. Please check [SQL DDL docs](sql_ddl#create-expression-index) for more details.
+3. Expression index is supported for unary and certain binary expressions. Please check [SQL DDL docs](sql_ddl.md#create-expression-index) for more details.
:::
The `expr` option is required for creating expression index, and it should be a valid Spark SQL function. Please check the syntax
@@ -577,7 +577,7 @@ Users can set table properties while creating a table. The important table prope
| Parameter Name | Default | Description |
|------------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types) |
+| type | cow | The table type to create. `type = 'cow'` creates a COPY-ON-WRITE table, while `type = 'mor'` creates a MERGE-ON-READ table. Same as `hoodie.datasource.write.table.type`. More details can be found [here](table_types.md) |
| primaryKey | uuid | The primary key field names of the table separated by commas. Same as `hoodie.datasource.write.recordkey.field`. If this config is ignored, hudi will auto-generate primary keys. If explicitly set, primary key generation will honor user configuration. |
| preCombineField | | The pre-combine field of the table. It is used for resolving the final version of the record among multiple versions. Generally, `event time` or another similar column will be used for ordering purposes. Hudi will be able to handle out-of-order data using the preCombine field value. |
@@ -588,7 +588,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control.md#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -839,7 +839,7 @@ WITH (
### Create Table in Non-Blocking Concurrency Control Mode
-The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control#non-blocking-concurrency-control).
+The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control.md#non-blocking-concurrency-control).
```sql
-- This is a datagen source that can generate records continuously
@@ -907,7 +907,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations.md#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-1.0.2/sql_dml.md b/website/versioned_docs/version-1.0.2/sql_dml.md
index 16db53dcd0ef..6b257144287c 100644
--- a/website/versioned_docs/version-1.0.2/sql_dml.md
+++ b/website/versioned_docs/version-1.0.2/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl.md) for creating Hudi tables using SQL.
### Insert Into
@@ -29,7 +29,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations.md#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -49,7 +49,7 @@ INSERT INTO hudi_cow_pt_tbl PARTITION(dt, hh) SELECT 1 AS id, 'a1' AS name, 1000
```
:::note Mapping to write operations
-Hudi offers flexibility in choosing the underlying [write operation](write_operations) of a `INSERT INTO` statement using
+Hudi offers flexibility in choosing the underlying [write operation](write_operations.md) of a `INSERT INTO` statement using
the `hoodie.spark.sql.insert.into.operation` configuration. Possible options include *"bulk_insert"* (large inserts), *"insert"* (with small file management),
and *"upsert"* (with deduplication/merging). If a precombine field is not set, *"insert"* is chosen as the default. For a table with preCombine field set,
*"upsert"* is chosen as the default operation.
@@ -409,7 +409,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control.md#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-1.0.2/sql_queries.md b/website/versioned_docs/version-1.0.2/sql_queries.md
index 7d70f7353805..f00bee112179 100644
--- a/website/versioned_docs/version-1.0.2/sql_queries.md
+++ b/website/versioned_docs/version-1.0.2/sql_queries.md
@@ -8,11 +8,11 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-Hudi stores and organizes data on storage while providing different ways of [querying](concepts#query-types), across a wide range of query engines.
+Hudi stores and organizes data on storage while providing different ways of [querying](concepts.md#query-types), across a wide range of query engines.
This page will show how to issue different queries and discuss any specific instructions for each query engine.
## Spark SQL
-The Spark [quickstart](quick-start-guide) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
+The Spark [quickstart](quick-start-guide.md) provides a good overview of how to use Spark SQL to query Hudi tables. This section will go into more advanced configurations and functionalities.
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
@@ -90,7 +90,7 @@ show the files scanned as 1 compared to 3 files scanned without index.
:::note
Please note in order to create secondary index:
-1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger#commit_time_ordering).
+1. The table must have a primary key and merge mode should be [COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
2. Record index must be enabled. This can be done by setting `hoodie.metadata.record.index.enable=true` and then creating `record_index`. Please note the example below.
:::
@@ -196,7 +196,7 @@ DROP INDEX partition_stats on hudi_indexed_table;
### Snapshot Query with Event Time Ordering
-Hudi supports different [record merge modes](record_merger) for merging the records from the same key. Event
+Hudi supports different [record merge modes](record_merger.md) for merging the records from the same key. Event
time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
event time ordering merge mode.
@@ -323,7 +323,7 @@ For e.g the table has received 10 million modifications across 1 million records
see all changes in a given time window and not just the latest values.
:::
-Please refer to [configurations](basic_configurations) section for the important configuration options.
+Please refer to [configurations](basic_configurations.md) section for the important configuration options.
:::note Incremental Query Checkpointing between Hudi 0.x and 1.0.
In Hudi 1.0, we switch the incremental and CDC query to used completion time, instead of instant time, to determine the
@@ -406,7 +406,7 @@ and the corresponding type column value.
## Flink SQL
Once the Flink Hudi tables have been registered to the Flink catalog, they can be queried using the Flink SQL. It supports all query types across both Hudi table types,
-relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide).
+relying on the custom Hudi input formats like Hive. Typically, notebook users and Flink SQL CLI users leverage flink sql for querying Hudi tables. Please add hudi-flink-bundle as described in the [Flink Quickstart](flink-quick-start-guide.md).
### Snapshot Query
diff --git a/website/versioned_docs/version-1.0.2/storage_layouts.md b/website/versioned_docs/version-1.0.2/storage_layouts.md
index 3b7911a15a1d..a88f584e3f9d 100644
--- a/website/versioned_docs/version-1.0.2/storage_layouts.md
+++ b/website/versioned_docs/version-1.0.2/storage_layouts.md
@@ -12,8 +12,8 @@ The following describes the general organization of files in storage for a Hudi
* Each slice contains a **_base file_** (parquet/orc/hfile) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) )
written by a commit that completed at a certain instant, along with set of **_log files_** (*.log.*) written by commits that completed
before the next base file's requested instant.
-* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
- file slices and [cleaning](cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction.md) action merges logs and base files to produce new
+ file slices and [cleaning](cleaning.md) action gets rid of unused/older file slices to reclaim space on the file system.
* All metadata including timeline, metadata table are stored in a special `.hoodie` directory under the base path.
diff --git a/website/versioned_docs/version-1.0.2/structure.md b/website/versioned_docs/version-1.0.2/structure.md
index 31281b68f71c..9f78f89f16f5 100644
--- a/website/versioned_docs/version-1.0.2/structure.md
+++ b/website/versioned_docs/version-1.0.2/structure.md
@@ -17,4 +17,4 @@ Hudi (pronounced βHoodieβ) ingests & manages storage of large analytical tab
By carefully managing how data is laid out in storage & how itβs exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like [PrestoDB](https://prestodb.io) & [Spark](https://spark.apache.org/sql/), while at the same time capable of being consumed incrementally from processing/ETL frameworks like [Hive](https://hive.apache.org/) & [Spark](https://spark.apache.org/docs/latest/) [...]
-Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide) for a demo.
+Hudi broadly consists of a self contained Spark library to build tables and integrations with existing query engines for data access. See [quickstart](quick-start-guide.md) for a demo.
diff --git a/website/versioned_docs/version-1.0.2/syncing_aws_glue_data_catalog.md b/website/versioned_docs/version-1.0.2/syncing_aws_glue_data_catalog.md
index b6f6c82a6c54..35f43a8af472 100644
--- a/website/versioned_docs/version-1.0.2/syncing_aws_glue_data_catalog.md
+++ b/website/versioned_docs/version-1.0.2/syncing_aws_glue_data_catalog.md
@@ -10,7 +10,7 @@ and send them to AWS Glue.
## Configurations
... 135 lines suppressed ...