You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by bh...@apache.org on 2024/12/11 22:31:37 UTC
(hudi) branch asf-site updated: [site] Fix broken links across versions (#12470)
This is an automated email from the ASF dual-hosted git repository.
bhavanisudha pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 43420a2e7b5 [site] Fix broken links across versions (#12470)
43420a2e7b5 is described below
commit 43420a2e7b5c2e026291cd041c74e220007f7a8d
Author: Bhavani Sudha Saktheeswaran <21...@users.noreply.github.com>
AuthorDate: Wed Dec 11 14:31:31 2024 -0800
[site] Fix broken links across versions (#12470)
---
README.md | 39 ++++++++++++-----
.../docs/0.8.0/concurrency_control/.index.html.swp | Bin 0 -> 4096 bytes
.../blog/2019-09-09-ingesting-database-changes.md | 2 +-
.../blog/2020-01-20-change-capture-using-aws.md | 2 +-
website/blog/2021-01-27-hudi-clustering-intro.md | 2 +-
website/blog/2021-03-01-hudi-file-sizing.md | 6 +--
.../blog/2021-08-16-kafka-custom-deserializer.md | 2 +-
website/docs/cli.md | 2 +-
website/docs/concurrency_control.md | 6 +--
website/docs/deployment.md | 2 +-
website/docs/faq.md | 14 +++----
website/docs/faq_general.md | 2 +-
website/docs/faq_table_services.md | 2 +-
website/docs/faq_writing_tables.md | 8 ++--
website/docs/file_sizing.md | 2 +-
website/docs/flink-quick-start-guide.md | 18 ++++----
website/docs/hudi_stack.md | 2 +-
website/docs/indexes.md | 12 +++---
website/docs/metadata.md | 4 +-
website/docs/metadata_indexing.md | 8 ++--
website/docs/precommit_validator.md | 2 +-
website/docs/procedures.md | 20 ++++-----
website/docs/querying_data.md | 2 +-
website/docs/quick-start-guide.md | 28 ++++++-------
website/docs/record_merger.md | 6 +--
website/docs/sql_ddl.md | 14 +++----
website/docs/sql_dml.md | 6 +--
website/docs/sql_queries.md | 2 +-
website/docs/storage_layouts.md | 4 +-
website/docs/use_cases.md | 8 ++--
website/docs/write_operations.md | 16 +++----
website/docs/writing_data.md | 4 +-
website/releases/release-0.11.0.md | 2 +-
website/releases/release-0.7.0.md | 2 +-
website/releases/release-0.9.0.md | 2 +-
website/releases/release-1.0.0-beta1.md | 4 +-
website/releases/release-1.0.0-beta2.md | 2 +-
.../versioned_docs/version-0.10.0/clustering.md | 2 +-
.../versioned_docs/version-0.10.0/compaction.md | 2 +-
.../version-0.10.0/concurrency_control.md | 10 ++---
.../versioned_docs/version-0.10.0/deployment.md | 6 +--
website/versioned_docs/version-0.10.0/faq.md | 2 +-
.../versioned_docs/version-0.10.0/file_sizing.md | 2 +-
.../versioned_docs/version-0.10.0/performance.md | 6 +--
.../versioned_docs/version-0.10.0/querying_data.md | 8 ++--
.../version-0.10.0/quick-start-guide.md | 4 +-
.../version-0.10.0/write_operations.md | 4 +-
.../versioned_docs/version-0.10.0/writing_data.md | 4 +-
.../versioned_docs/version-0.10.1/clustering.md | 2 +-
.../versioned_docs/version-0.10.1/compaction.md | 2 +-
.../version-0.10.1/concurrency_control.md | 10 ++---
.../versioned_docs/version-0.10.1/deployment.md | 6 +--
website/versioned_docs/version-0.10.1/faq.md | 2 +-
.../versioned_docs/version-0.10.1/file_sizing.md | 2 +-
.../versioned_docs/version-0.10.1/performance.md | 6 +--
.../versioned_docs/version-0.10.1/querying_data.md | 8 ++--
.../version-0.10.1/quick-start-guide.md | 4 +-
.../versioned_docs/version-0.10.1/tuning-guide.md | 2 +-
.../version-0.10.1/write_operations.md | 4 +-
.../versioned_docs/version-0.10.1/writing_data.md | 4 +-
.../versioned_docs/version-0.11.0/compaction.md | 2 +-
.../versioned_docs/version-0.11.0/deployment.md | 2 +-
website/versioned_docs/version-0.11.0/faq.md | 2 +-
.../versioned_docs/version-0.11.0/file_sizing.md | 2 +-
.../versioned_docs/version-0.11.0/querying_data.md | 2 +-
.../versioned_docs/version-0.11.0/tuning-guide.md | 2 +-
.../version-0.11.0/write_operations.md | 2 +-
.../versioned_docs/version-0.11.1/compaction.md | 2 +-
.../versioned_docs/version-0.11.1/deployment.md | 2 +-
website/versioned_docs/version-0.11.1/faq.md | 2 +-
.../versioned_docs/version-0.11.1/file_sizing.md | 2 +-
.../versioned_docs/version-0.11.1/querying_data.md | 2 +-
.../versioned_docs/version-0.11.1/tuning-guide.md | 2 +-
.../version-0.11.1/write_operations.md | 2 +-
.../versioned_docs/version-0.12.0/compaction.md | 2 +-
.../versioned_docs/version-0.12.0/deployment.md | 2 +-
website/versioned_docs/version-0.12.0/faq.md | 2 +-
.../versioned_docs/version-0.12.0/file_sizing.md | 2 +-
.../version-0.12.0/flink-quick-start-guide.md | 2 +-
.../versioned_docs/version-0.12.0/querying_data.md | 2 +-
.../versioned_docs/version-0.12.0/tuning-guide.md | 2 +-
.../version-0.12.0/write_operations.md | 2 +-
.../versioned_docs/version-0.12.1/compaction.md | 2 +-
.../versioned_docs/version-0.12.1/deployment.md | 2 +-
website/versioned_docs/version-0.12.1/faq.md | 2 +-
.../versioned_docs/version-0.12.1/file_sizing.md | 2 +-
.../versioned_docs/version-0.12.1/querying_data.md | 2 +-
.../versioned_docs/version-0.12.1/tuning-guide.md | 2 +-
.../version-0.12.1/write_operations.md | 2 +-
.../versioned_docs/version-0.12.2/compaction.md | 2 +-
.../versioned_docs/version-0.12.2/deployment.md | 2 +-
website/versioned_docs/version-0.12.2/faq.md | 2 +-
.../versioned_docs/version-0.12.2/file_sizing.md | 2 +-
.../version-0.12.2/flink-quick-start-guide.md | 2 +-
.../versioned_docs/version-0.12.2/querying_data.md | 2 +-
.../version-0.12.2/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.12.2/tuning-guide.md | 2 +-
.../version-0.12.2/write_operations.md | 2 +-
.../versioned_docs/version-0.12.3/compaction.md | 2 +-
.../versioned_docs/version-0.12.3/deployment.md | 2 +-
website/versioned_docs/version-0.12.3/faq.md | 2 +-
.../versioned_docs/version-0.12.3/file_sizing.md | 2 +-
.../version-0.12.3/flink-quick-start-guide.md | 2 +-
.../versioned_docs/version-0.12.3/querying_data.md | 2 +-
.../version-0.12.3/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.12.3/tuning-guide.md | 2 +-
.../version-0.12.3/write_operations.md | 2 +-
.../versioned_docs/version-0.13.0/compaction.md | 2 +-
.../versioned_docs/version-0.13.0/deployment.md | 2 +-
website/versioned_docs/version-0.13.0/faq.md | 2 +-
.../versioned_docs/version-0.13.0/file_sizing.md | 2 +-
.../version-0.13.0/flink-quick-start-guide.md | 2 +-
.../versioned_docs/version-0.13.0/querying_data.md | 2 +-
.../version-0.13.0/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.13.0/tuning-guide.md | 2 +-
.../version-0.13.0/write_operations.md | 2 +-
.../versioned_docs/version-0.13.1/compaction.md | 2 +-
.../versioned_docs/version-0.13.1/deployment.md | 2 +-
website/versioned_docs/version-0.13.1/faq.md | 2 +-
.../versioned_docs/version-0.13.1/file_sizing.md | 2 +-
.../version-0.13.1/flink-quick-start-guide.md | 6 +--
.../version-0.13.1/quick-start-guide.md | 2 +-
.../version-0.13.1/record_payload.md | 2 +-
.../versioned_docs/version-0.13.1/tuning-guide.md | 2 +-
.../version-0.13.1/write_operations.md | 2 +-
.../versioned_docs/version-0.14.0/deployment.md | 2 +-
website/versioned_docs/version-0.14.0/faq.md | 4 +-
.../version-0.14.0/flink-quick-start-guide.md | 6 +--
.../version-0.14.0/quick-start-guide.md | 4 +-
.../version-0.14.0/record_payload.md | 2 +-
.../version-0.14.0/write_operations.md | 2 +-
website/versioned_docs/version-0.14.1/cli.md | 2 +-
.../version-0.14.1/concurrency_control.md | 2 +-
.../versioned_docs/version-0.14.1/deployment.md | 2 +-
.../version-0.14.1/faq_writing_tables.md | 6 +--
.../versioned_docs/version-0.14.1/file_layouts.md | 4 +-
.../versioned_docs/version-0.14.1/file_sizing.md | 2 +-
.../version-0.14.1/flink-quick-start-guide.md | 18 ++++----
website/versioned_docs/version-0.14.1/indexing.md | 4 +-
.../version-0.14.1/metadata_indexing.md | 4 +-
.../versioned_docs/version-0.14.1/procedures.md | 46 ++++++++++-----------
.../versioned_docs/version-0.14.1/querying_data.md | 2 +-
.../version-0.14.1/quick-start-guide.md | 30 +++++++-------
.../version-0.14.1/record_payload.md | 2 +-
website/versioned_docs/version-0.14.1/sql_ddl.md | 6 +--
website/versioned_docs/version-0.14.1/sql_dml.md | 4 +-
website/versioned_docs/version-0.14.1/use_cases.md | 12 +++---
.../version-0.14.1/write_operations.md | 16 +++----
.../versioned_docs/version-0.14.1/writing_data.md | 16 +++----
website/versioned_docs/version-0.15.0/cli.md | 2 +-
.../version-0.15.0/concurrency_control.md | 2 +-
.../versioned_docs/version-0.15.0/deployment.md | 2 +-
website/versioned_docs/version-0.15.0/faq.md | 2 +-
.../versioned_docs/version-0.15.0/faq_general.md | 2 +-
.../version-0.15.0/faq_table_services.md | 2 +-
.../version-0.15.0/faq_writing_tables.md | 8 ++--
.../versioned_docs/version-0.15.0/file_layouts.md | 4 +-
.../versioned_docs/version-0.15.0/file_sizing.md | 2 +-
.../version-0.15.0/flink-quick-start-guide.md | 18 ++++----
.../versioned_docs/version-0.15.0/hudi_stack.md | 2 +-
website/versioned_docs/version-0.15.0/indexing.md | 4 +-
.../version-0.15.0/metadata_indexing.md | 4 +-
.../version-0.15.0/precommit_validator.md | 2 +-
.../versioned_docs/version-0.15.0/procedures.md | 46 ++++++++++-----------
.../versioned_docs/version-0.15.0/querying_data.md | 2 +-
.../version-0.15.0/quick-start-guide.md | 26 ++++++------
.../version-0.15.0/record_payload.md | 2 +-
website/versioned_docs/version-0.15.0/sql_ddl.md | 4 +-
website/versioned_docs/version-0.15.0/sql_dml.md | 6 +--
website/versioned_docs/version-0.15.0/use_cases.md | 12 +++---
.../version-0.15.0/write_operations.md | 16 +++----
.../versioned_docs/version-0.15.0/writing_data.md | 4 +-
.../versioned_docs/version-0.5.0/querying_data.md | 2 +-
.../version-0.5.0/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.5.0/writing_data.md | 6 +--
website/versioned_docs/version-0.5.1/deployment.md | 6 +--
.../versioned_docs/version-0.5.1/querying_data.md | 2 +-
.../version-0.5.1/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.5.1/writing_data.md | 6 +--
website/versioned_docs/version-0.5.2/deployment.md | 6 +--
.../versioned_docs/version-0.5.2/querying_data.md | 4 +-
.../version-0.5.2/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.5.2/writing_data.md | 6 +--
website/versioned_docs/version-0.5.3/deployment.md | 6 +--
.../versioned_docs/version-0.5.3/querying_data.md | 4 +-
.../version-0.5.3/quick-start-guide.md | 4 +-
.../versioned_docs/version-0.5.3/writing_data.md | 6 +--
website/versioned_docs/version-0.6.0/deployment.md | 6 +--
.../versioned_docs/version-0.6.0/querying_data.md | 4 +-
.../version-0.6.0/quick-start-guide.md | 4 +-
.../versioned_docs/version-0.6.0/writing_data.md | 6 +--
website/versioned_docs/version-0.7.0/deployment.md | 6 +--
.../versioned_docs/version-0.7.0/querying_data.md | 4 +-
.../version-0.7.0/quick-start-guide.md | 4 +-
.../versioned_docs/version-0.7.0/writing_data.md | 6 +--
.../version-0.8.0/concurrency_control.md | 10 ++---
website/versioned_docs/version-0.8.0/deployment.md | 6 +--
.../versioned_docs/version-0.8.0/querying_data.md | 4 +-
.../version-0.8.0/quick-start-guide.md | 2 +-
.../versioned_docs/version-0.8.0/writing_data.md | 6 +--
.../version-0.9.0/concurrency_control.md | 10 ++---
website/versioned_docs/version-0.9.0/deployment.md | 6 +--
.../versioned_docs/version-0.9.0/querying_data.md | 4 +-
.../version-0.9.0/quick-start-guide.md | 4 +-
.../versioned_docs/version-0.9.0/writing_data.md | 6 +--
205 files changed, 524 insertions(+), 505 deletions(-)
diff --git a/README.md b/README.md
index 9b011e8c716..5686cf42d7e 100644
--- a/README.md
+++ b/README.md
@@ -129,16 +129,35 @@ versioned_sidebars/version-0.7.0-sidebars.json
```
### Linking docs
-
-- Remember to include the `.md` extension.
-- Files will be linked to correct corresponding version.
-- Relative paths work as well.
-
-```md
-The [@hello](hello.md#paginate) document is great!
-
-See the [Tutorial](../getting-started/tutorial.md) for more info.
-```
+Relative paths work well. - Files will be linked to correct corresponding version.
+ - PREFER RELATIVE PATHS to be consistent with linking.
+ - **Good Example of linking.**
+ For ex say we are updating a 0.12.0 version doc which is older.
+ ```md
+ A [callback notification](writing_data#commit-notifications) is exposed
+ ```
+ This automatically resolves to /docs/0.12.0/writing_data#commit-notifications.
+ - **Bad example of linking.**
+ For ex say we are updating a 0.12.0 version doc which is older.
+ ```md
+ A [callback notification](/docs/writing_data#commit-notifications) is exposed
+ ```
+ This will resolve to the most recent release, specifically /docs/writing_data#commit-notifications . We do not want a 0.12.0 doc page to point to a page from a later release.
+ - DO NOT use next version when linking.
+ - Good Example of linking when you are working on unreleased version (from next version).
+ ```md
+ Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
+ file slices and [cleaning](cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+ ```
+ This automatically resolves to /docs/next/compaction and /docs/next/cleaning pages.
+
+ - Bad Example of linking when you are working on unreleased version (from next version).
+ ```md
+ Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/next/compaction) action merges logs and base files to produce new
+ file slices and [cleaning](/docs/next/cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+ ```
+ Even though it directly points to /docs/next which is intended target, this accumulates as tech debt when this copy of docs gets released, we will hav a older doc always pointing to /docs/next/
+
## Versions
diff --git a/content/docs/0.8.0/concurrency_control/.index.html.swp b/content/docs/0.8.0/concurrency_control/.index.html.swp
new file mode 100644
index 00000000000..122ce09e20f
Binary files /dev/null and b/content/docs/0.8.0/concurrency_control/.index.html.swp differ
diff --git a/website/blog/2019-09-09-ingesting-database-changes.md b/website/blog/2019-09-09-ingesting-database-changes.md
index 2c8b068e5a2..79373fd7768 100644
--- a/website/blog/2019-09-09-ingesting-database-changes.md
+++ b/website/blog/2019-09-09-ingesting-database-changes.md
@@ -44,5 +44,5 @@ inputDataset.write.format("org.apache.hudi”)
.save("/path/on/dfs");
```
-Alternatively, you can also use the Hudi [DeltaStreamer](https://hudi.apache.org/writing_data#deltastreamer) tool with the DFSSource.
+Alternatively, you can also use the Hudi [DeltaStreamer](https://hudi.apache.org/docs/hoodie_streaming_ingestion#hudi-streamer) tool with the DFSSource.
diff --git a/website/blog/2020-01-20-change-capture-using-aws.md b/website/blog/2020-01-20-change-capture-using-aws.md
index c7ff93ec1b8..29b2589174d 100644
--- a/website/blog/2020-01-20-change-capture-using-aws.md
+++ b/website/blog/2020-01-20-change-capture-using-aws.md
@@ -20,7 +20,7 @@ In this blog, we will build an end-end solution for capturing changes from a MyS
We can break up the problem into two pieces.
1. **Extracting change logs from MySQL** : Surprisingly, this is still a pretty tricky problem to solve and often Hudi users get stuck here. Thankfully, at-least for AWS users, there is a [Database Migration service](https://aws.amazon.com/dms/) (DMS for short), that does this change capture and uploads them as parquet files on S3
-2. **Applying these change logs to your data lake table** : Once there are change logs in some form, the next step is to apply them incrementally to your table. This mundane task can be fully automated using the Hudi [DeltaStreamer](http://hudi.apache.org/docs/writing_data#deltastreamer) tool.
+2. **Applying these change logs to your data lake table** : Once there are change logs in some form, the next step is to apply them incrementally to your table. This mundane task can be fully automated using the Hudi [DeltaStreamer](http://hudi.apache.org/docs/hoodie_streaming_ingestion#hudi-streamer) tool.
diff --git a/website/blog/2021-01-27-hudi-clustering-intro.md b/website/blog/2021-01-27-hudi-clustering-intro.md
index f1af4433e5e..d227d729ce3 100644
--- a/website/blog/2021-01-27-hudi-clustering-intro.md
+++ b/website/blog/2021-01-27-hudi-clustering-intro.md
@@ -17,7 +17,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## Clustering Architecture
-At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations#compactionSmallFileSize) to `0` to force new data [...]
+At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) to `0` to force new [...]
diff --git a/website/blog/2021-03-01-hudi-file-sizing.md b/website/blog/2021-03-01-hudi-file-sizing.md
index 304c2aaaabb..808688b3733 100644
--- a/website/blog/2021-03-01-hudi-file-sizing.md
+++ b/website/blog/2021-03-01-hudi-file-sizing.md
@@ -36,9 +36,9 @@ For illustration purposes, we are going to consider only COPY_ON_WRITE table.
Configs of interest before we dive into the algorithm:
-- [Max file size](/docs/configurations#limitFileSize): Max size for a given data file. Hudi will try to maintain file sizes to this configured value <br/>
-- [Soft file limit](/docs/configurations#compactionSmallFileSize): Max file size below which a given data file is considered to a small file <br/>
-- [Insert split size](/docs/configurations#insertSplitSize): Number of inserts grouped for a single partition. This value should match
+- [Max file size](/docs/configurations#hoodieparquetmaxfilesize): Max size for a given data file. Hudi will try to maintain file sizes to this configured value <br/>
+- [Soft file limit](/docs/configurations#hoodieparquetsmallfilelimit): Max file size below which a given data file is considered to a small file <br/>
+- [Insert split size](/docs/configurations#hoodiecopyonwriteinsertsplitsize): Number of inserts grouped for a single partition. This value should match
the number of records in a single file (you can determine based on max file size and per record size)
For instance, if your first config value is 120MB and 2nd config value is set to 100MB, any file whose size is < 100MB
diff --git a/website/blog/2021-08-16-kafka-custom-deserializer.md b/website/blog/2021-08-16-kafka-custom-deserializer.md
index c8146b6343d..7ed4bf3e03a 100644
--- a/website/blog/2021-08-16-kafka-custom-deserializer.md
+++ b/website/blog/2021-08-16-kafka-custom-deserializer.md
@@ -18,7 +18,7 @@ In our case a Confluent schema registry is used to maintain the schema and as sc
<!--truncate-->
## What do we want to achieve?
-We have multiple instances of DeltaStreamer running, consuming many topics with different schemas ingesting to multiple Hudi tables. Deltastreamer is a utility in Hudi to assist in ingesting data from multiple sources like DFS, kafka, etc into Hudi. If interested, you can read more about DeltaStreamer tool [here](https://hudi.apache.org/docs/writing_data#deltastreamer)
+We have multiple instances of DeltaStreamer running, consuming many topics with different schemas ingesting to multiple Hudi tables. Deltastreamer is a utility in Hudi to assist in ingesting data from multiple sources like DFS, kafka, etc into Hudi. If interested, you can read more about DeltaStreamer tool [here](https://hudi.apache.org/docs/hoodie_streaming_ingestion#hudi-streamer)
Ideally every topic should be able to evolve the schema to match new business requirements. Producers start producing data with a new schema version and the DeltaStreamer picks up the new schema and ingests the data with the new schema. For this to work, we run our DeltaStreamer instances with the latest schema version available from the Schema Registry to ensure that we always use the freshest schema with all attributes.
A prerequisites is that all the mentioned Schema evolutions must be `BACKWARD_TRANSITIVE` compatible (see [Schema Evolution and Compatibility of Avro Schema changes](https://docs.confluent.io/platform/current/schema-registry/avro.html). This ensures that every record in the kafka topic can always be read using the latest schema.
diff --git a/website/docs/cli.md b/website/docs/cli.md
index 1c30b9b6fa6..7cc4cdd92b0 100644
--- a/website/docs/cli.md
+++ b/website/docs/cli.md
@@ -578,7 +578,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](/docs/next/disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
To trigger savepoint for a hudi table
```java
diff --git a/website/docs/concurrency_control.md b/website/docs/concurrency_control.md
index 8550888e734..d9867be88a8 100644
--- a/website/docs/concurrency_control.md
+++ b/website/docs/concurrency_control.md
@@ -8,7 +8,7 @@ last_modified_at: 2021-03-19T15:59:57-04:00
---
Concurrency control defines how different writers/readers/table services coordinate access to a Hudi table. Hudi ensures atomic writes, by way of publishing commits atomically to the timeline,
stamped with an instant time that denotes the time at which the action is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between
-writer processes that issue [write operations](/docs/next/write_operations) and table services that (re)write data/metadata to optimize/perform bookkeeping and
+writer processes that issue [write operations](write_operations) and table services that (re)write data/metadata to optimize/perform bookkeeping and
readers (that execute queries and read data).
Hudi provides
@@ -23,7 +23,7 @@ We’ll also describe ways to ingest data into a Hudi Table from multiple writer
## Distributed Locking
A pre-requisite for distributed co-ordination in Hudi, like many other distributed database systems is a distributed lock provider, that different processes can use to plan, schedule and
-execute actions on the Hudi timeline in a concurrent fashion. Locks are also used to [generate TrueTime](/docs/next/timeline#truetime-generation), as discussed before.
+execute actions on the Hudi timeline in a concurrent fashion. Locks are also used to [generate TrueTime](timeline#truetime-generation), as discussed before.
External locking is typically used in conjunction with optimistic concurrency control
because it provides a way to prevent conflicts that might occur when two or more transactions (commits in our case) attempt to modify the same resource concurrently.
@@ -204,7 +204,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](/docs/next/sql_dml#non-blocking-concurrency-control-experimental).
+writers [here](sql_dml#non-blocking-concurrency-control-experimental).
## Early conflict Detection
diff --git a/website/docs/deployment.md b/website/docs/deployment.md
index 9bafde59c46..7785f4ceaca 100644
--- a/website/docs/deployment.md
+++ b/website/docs/deployment.md
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Batch Writes](/docs/next/writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.i [...]
+As described in [Batch Writes](writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.d [...]
Here is an example invocation using spark datasource
diff --git a/website/docs/faq.md b/website/docs/faq.md
index 1378839b81c..26c3eb50d21 100644
--- a/website/docs/faq.md
+++ b/website/docs/faq.md
@@ -6,10 +6,10 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](/docs/next/faq_general)
-- [Design & Concepts](/docs/next/faq_design_and_concepts)
-- [Writing Tables](/docs/next/faq_writing_tables)
-- [Reading Tables](/docs/next/faq_reading_tables)
-- [Table Services](/docs/next/faq_table_services)
-- [Storage](/docs/next/faq_storage)
-- [Integrations](/docs/next/faq_integrations)
+- [General](faq_general)
+- [Design & Concepts](faq_design_and_concepts)
+- [Writing Tables](faq_writing_tables)
+- [Reading Tables](faq_reading_tables)
+- [Table Services](faq_table_services)
+- [Storage](faq_storage)
+- [Integrations](faq_integrations)
diff --git a/website/docs/faq_general.md b/website/docs/faq_general.md
index 61b6c12a4b5..9f0a6c7d515 100644
--- a/website/docs/faq_general.md
+++ b/website/docs/faq_general.md
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](/docs/next/writing_data) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
diff --git a/website/docs/faq_table_services.md b/website/docs/faq_table_services.md
index 0ca730094e4..7ff398687e3 100644
--- a/website/docs/faq_table_services.md
+++ b/website/docs/faq_table_services.md
@@ -50,6 +50,6 @@ Hudi runs cleaner to remove old file versions as part of writing data either in
Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
-be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](/docs/next/platform_services_post_commit_callback)
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback)
for details
diff --git a/website/docs/faq_writing_tables.md b/website/docs/faq_writing_tables.md
index bed07a16e57..2374006d955 100644
--- a/website/docs/faq_writing_tables.md
+++ b/website/docs/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](/docs/hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasour [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](/docs/hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasour [...]
### How is a Hudi writer job deployed?
@@ -68,7 +68,7 @@ As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github
### How do I delete records in the dataset using Hudi?
-GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](/docs/next/writing_data#deletes).
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data#deletes).
### Should I need to worry about deleting all copies of the records in case of duplicates?
@@ -147,7 +147,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size base [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size base [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
@@ -183,7 +183,7 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](/docs/concurrency_control#supported-concurrency-controls) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-com [...]
+Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
### Can single-writer inserts have duplicates?
diff --git a/website/docs/file_sizing.md b/website/docs/file_sizing.md
index c637a5a630c..62ad0f7a432 100644
--- a/website/docs/file_sizing.md
+++ b/website/docs/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](/docs/next/cleaning)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](cleaning)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/docs/flink-quick-start-guide.md b/website/docs/flink-quick-start-guide.md
index 0ab2322d766..1cfda067c71 100644
--- a/website/docs/flink-quick-start-guide.md
+++ b/website/docs/flink-quick-start-guide.md
@@ -449,19 +449,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](/docs/next/flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/next/flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/next/ingestion_flink#cdc-ingestion), [Bulk Insert](/docs/next/ingestion_flink#bulk-insert), [Index Bootstrap](/docs/next/ingestion_flink#index-bootstrap), [Changelog Mode](/docs/next/ingestion_flink#changelog-mode) and [Append Mode](/docs/next/ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](/docs/next/sql_dml#non-blocking-conc [...]
-- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/sql_queries#streaming-query) and [Incremental Query](/docs/sql_queries#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/next/flink_tuning#memory-optimization) and [Write Rate Limit](/docs/next/flink_tuning#write-rate-limit).
+- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink#cdc-ingestion), [Bulk Insert](ingestion_flink#bulk-insert), [Index Bootstrap](ingestion_flink#index-bootstrap), [Changelog Mode](ingestion_flink#changelog-mode) and [Append Mode](ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](/docs/sql_queries#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/querying_data#prestodb).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](/docs/sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](/docs/next/timeline) – How Hudi manages transactions and other table services
- - [Hudi Storage Layout](/docs/next/storage_layouts) - How the files are laid out on storage
- - [Hudi Table Types](/docs/next/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](/docs/next/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline) – How Hudi manages transactions and other table services
+ - [Hudi Storage Layout](storage_layouts) - How the files are laid out on storage
+ - [Hudi Table Types](table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
diff --git a/website/docs/hudi_stack.md b/website/docs/hudi_stack.md
index ab2408f4316..59517ede41d 100644
--- a/website/docs/hudi_stack.md
+++ b/website/docs/hudi_stack.md
@@ -157,7 +157,7 @@ Platform services offer functionality that is specific to data and workloads, an
Services, like [Hudi Streamer](./hoodie_streaming_ingestion#hudi-streamer) (or its Flink counterpart), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various
formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (including Confluent), and
deduplication of data. Hudi Streamer also offers features for backfills, one-off runs, and continuous mode operation with Spark/Flink streaming writers. Additionally,
-Hudi provides tools for [snapshotting](./snapshot_exporter) and incrementally [exporting](./snapshot_exporter#examples) Hudi tables, importing new tables, and [post-commit callback](/docs/next/platform_services_post_commit_callback) for analytics or
+Hudi provides tools for [snapshotting](./snapshot_exporter) and incrementally [exporting](./snapshot_exporter#examples) Hudi tables, importing new tables, and [post-commit callback](platform_services_post_commit_callback) for analytics or
workflow management, enhancing the deployment of production-grade incremental pipelines. Apart from these services, Hudi also provides broad support for different
catalogs such as [Hive Metastore](./syncing_metastore), [AWS Glue](./syncing_aws_glue_data_catalog/), [Google BigQuery](./gcp_bigquery), [DataHub](./syncing_datahub), etc. that allows syncing of Hudi tables to be queried by
interactive engines such as Trino and Presto.
diff --git a/website/docs/indexes.md b/website/docs/indexes.md
index 512242ba811..c2284f2d473 100644
--- a/website/docs/indexes.md
+++ b/website/docs/indexes.md
@@ -19,8 +19,8 @@ Only clustering or cross-partition updates that are implemented as deletes + ins
file group at any completed instant on the timeline.
## Need for indexing
-For [Copy-On-Write tables](/docs/next/table_types#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](/docs/next/table_types#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
+For [Copy-On-Write tables](table_types#copy-on-write-table), indexing enables fast upsert/delete operations, by avoiding the need to join against the entire dataset to determine which files to rewrite.
+For [Merge-On-Read tables](table_types#merge-on-read-table), indexing allows Hudi to bound the amount of change records any given base file needs to be merged against. Specifically, a given base file needs to merged
only against updates for records that are part of that base file.
@@ -28,7 +28,7 @@ only against updates for records that are part of that base file.
In contrast,
- Designs without an indexing component (e.g: [Apache Hive/Apache Iceberg](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)) end up having to merge all the base files against all incoming updates/delete records
- (10-100x more [read amplification](/docs/next/table_types#comparison)).
+ (10-100x more [read amplification](table_types#comparison)).
- Designs that implement heavily write-optimized OLTP data structures like LSM trees do not require an indexing component. But they perform poorly scan heavy workloads
against cloud storage making them unsuitable for serving analytical queries.
@@ -42,8 +42,8 @@ implemented by enhancing the metadata table with the flexibility to extend to ne
along with an [asynchronous index](https://hudi.apache.org/docs/metadata_indexing/#setup-async-indexing) building
Hudi supports a multi-modal index by augmenting the metadata table with the capability to incorporate new types of indexes, complemented by an
-asynchronous mechanism for [index construction](/docs/next/metadata_indexing). This enhancement supports a range of indexes within
-the [metadata table](/docs/next/metadata#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
+asynchronous mechanism for [index construction](metadata_indexing). This enhancement supports a range of indexes within
+the [metadata table](metadata#metadata-table), significantly improving the efficiency of both writing to and reading from the table.
<p align = "center">Figure: Indexes in Hudi</p>
@@ -68,7 +68,7 @@ the [metadata table](/docs/next/metadata#metadata-table), significantly improvin
An [expression index](https://github.com/apache/hudi/blob/3789840be3d041cbcfc6b24786740210e4e6d6ac/rfc/rfc-63/rfc-63.md) is an index on a function of a column. If a query has a predicate on a function of a column, the expression index can
be used to speed up the query. Expression index is stored in *expr_index_* prefixed partitions (one for each
expression index) under metadata table. Expression index can be created using SQL syntax. Please checkout SQL DDL
- docs [here](/docs/next/sql_ddl#create-functional-index-experimental) for more details.
+ docs [here](sql_ddl#create-expression-index) for more details.
### Secondary Index
diff --git a/website/docs/metadata.md b/website/docs/metadata.md
index fb79f19799a..0295489e348 100644
--- a/website/docs/metadata.md
+++ b/website/docs/metadata.md
@@ -62,7 +62,7 @@ Following are the different types of metadata currently supported.
```
-To try out these features, refer to the [SQL guide](/docs/next/sql_ddl#create-partition-stats-and-secondary-index-experimental).
+To try out these features, refer to the [SQL guide](sql_ddl#create-partition-stats-index).
## Metadata Tracking on Writers
@@ -153,7 +153,7 @@ process which cannot rely on the in-process lock provider.
### Deployment Model C: Multi-writer
-If your current deployment model is [multi-writer](/docs/concurrency_control#model-c-multi-writer) along with a lock
+If your current deployment model is [multi-writer](concurrency_control#full-on-multi-writer--async-table-services) along with a lock
provider and other required configs set for every writer as follows, there is no additional configuration required. You
can bring up the writers sequentially after stopping the writers for enabling metadata table. Applying the proper
configurations to only partial writers leads to loss of data from the inconsistent writer. So, ensure you enable
diff --git a/website/docs/metadata_indexing.md b/website/docs/metadata_indexing.md
index ee0609965fb..d1978c1e486 100644
--- a/website/docs/metadata_indexing.md
+++ b/website/docs/metadata_indexing.md
@@ -31,7 +31,7 @@ asynchronous indexing. To learn more about the design of asynchronous indexing f
## Index Creation Using SQL
Currently indexes like secondary index, expression index and record index can be created using SQL create index command.
-For more information on these indexes please refer [metadata section](/docs/next/metadata/#types-of-table-metadata)
+For more information on these indexes please refer [metadata section](metadata/#types-of-table-metadata)
:::note
Please note in order to create secondary index:
@@ -54,7 +54,7 @@ CREATE INDEX idx_column_ts ON hudi_indexed_table USING column_stats(ts) OPTIONS(
CREATE INDEX idx_bloom_driver ON hudi_indexed_table USING bloom_filters(driver) OPTIONS(expr='identity');
```
-For more information on index creation using SQL refer [SQL DDL](/docs/next/sql_ddl#create-index)
+For more information on index creation using SQL refer [SQL DDL](sql_ddl#create-index)
## Index Creation Using Datasource
@@ -182,8 +182,8 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](/docs/next/metadata/#types-of-table-metadata) along with [configs](/docs/next/metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
-to enable them. The full set of metadata configurations can be explored [here](/docs/next/configurations/#Metadata-Configs).
+metadata table can be explored [here](indexes#multi-modal-indexing) along with [configs](metadata#metadata-tracking-on-writers)
+to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
Enabling the metadata table and configuring a lock provider are the prerequisites for using async indexer. Checkout a sample
diff --git a/website/docs/precommit_validator.md b/website/docs/precommit_validator.md
index 5e13fca3dc0..d5faf61057d 100644
--- a/website/docs/precommit_validator.md
+++ b/website/docs/precommit_validator.md
@@ -91,7 +91,7 @@ void validateRecordsBeforeAndAfter(Dataset<Row> before,
```
## Additional Monitoring with Notifications
-Hudi offers a [commit notification service](/docs/next/platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
+Hudi offers a [commit notification service](platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
The commit notification service can be combined with pre-commit validators to send a notification when a commit fails a validation. This is possible by passing details about the validation as a custom value to the HTTP endpoint.
diff --git a/website/docs/procedures.md b/website/docs/procedures.md
index 1dbeb899b14..19d65668011 100644
--- a/website/docs/procedures.md
+++ b/website/docs/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](/docs/next/configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](/docs/next/configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](/docs/next/configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](/docs/next/configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](/docs/next/configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](/docs/next/configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1535,10 +1535,10 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](/docs/next/configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the ma [...]
-| [retain_commits](/docs/next/configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](/docs/next/configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured n [...]
-| [file_versions_retained](/docs/next/configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
+| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
+| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
| [trigger_strategy](/docs/next/configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
| [trigger_max_commits](/docs/next/configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
| [options](/docs/next/configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
diff --git a/website/docs/querying_data.md b/website/docs/querying_data.md
index 31069822df7..83a03a4a112 100644
--- a/website/docs/querying_data.md
+++ b/website/docs/querying_data.md
@@ -7,7 +7,7 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](/docs/next/sql_ddl), [SQL DML](/docs/next/sql_dml), [SQL Queries](/docs/next/sql_queries) and [Procedures](/docs/next/procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
:::
Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](/docs/concepts#query-types).
diff --git a/website/docs/quick-start-guide.md b/website/docs/quick-start-guide.md
index 7c4c2e8077d..4ddb4005df3 100644
--- a/website/docs/quick-start-guide.md
+++ b/website/docs/quick-start-guide.md
@@ -257,7 +257,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](/docs/next/sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
</TabItem>
@@ -301,7 +301,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](/docs/next/write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -334,7 +334,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](/docs/next/write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -343,7 +343,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](/docs/next/sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -455,7 +455,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](/docs/next/sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -485,7 +485,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](/docs/next/concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -1264,7 +1264,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](/docs/next/write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1276,8 +1276,8 @@ Hudi also uses this mechanism to support out-of-order data arrival into a table,
For e.g. using a _created_at_ timestamp field as the precombine field will prevent older versions of a record from overwriting newer ones or being exposed to queries, even
if they are written at a later commit time to the table. This is one of the key features, that makes Hudi, best suited for dealing with streaming data.
-To enable different merge semantics, Hudi supports [merge modes](/docs/next/record_merger). Commit time and event time based merge modes are supported out of the box.
-Users can also define their own custom merge strategies, see [here](/docs/next/sql_ddl#create-table-with-record-merge-mode).
+To enable different merge semantics, Hudi supports [merge modes](record_merger). Commit time and event time based merge modes are supported out of the box.
+Users can also define their own custom merge strategies, see [here](sql_ddl#create-table-with-record-merge-mode).
<Tabs
groupId="programming-language"
@@ -1340,12 +1340,12 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](/docs/next/migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](/docs/next/sql_ddl) and [Spark SQL DML](/docs/next/sql_dml) reference guides.
-For alter table commands, check out [this](/docs/next/sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](/docs/next/procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
+For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
### Streaming workloads
@@ -1355,9 +1355,9 @@ Hudi provides industry-leading performance and functionality for streaming data.
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](/docs/next/writing_tables_streaming_writes#spark-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes#spark-streaming) for more.
-Check out more information on [modeling data in Hudi](/docs/next/faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](/docs/writing_data) and [streaming writes](/docs/next/writing_tables_streaming_writes).
+Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](/docs/writing_data) and [streaming writes](writing_tables_streaming_writes).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
diff --git a/website/docs/record_merger.md b/website/docs/record_merger.md
index d98a5fc462a..378c5575ad1 100644
--- a/website/docs/record_merger.md
+++ b/website/docs/record_merger.md
@@ -6,7 +6,7 @@ toc_min_heading_level: 2
toc_max_heading_level: 4
---
-Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](/docs/next/timeline#ordering-of-actions) section.
+Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](timeline#ordering-of-actions) section.
To provide users full-fledged support for stream processing, Hudi goes all the way making the storage engine and the underlying storage format
understand how to merge changes to the same record key, that may arrive even in different order at different times. With the rise of mobile applications
and IoT, these scenarios have become the normal than an exception. For e.g. a social networking application uploading user events several hours after they happened,
@@ -54,7 +54,7 @@ With event time ordering, the merging picks the record with the highest value on
In the example above, two microservices product change records about orders at different times, that can arrive out-of-order. As color coded,
this can lead to application-level inconsistent states in the table if simply merged in commit time order like a cancelled order being re-created or
a paid order moved back to just created state expecting payment again. Event time ordering helps by ignoring older state changes that arrive late and
-avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](/docs/next/concurrency_control#non-blocking-concurrency-control-mode),
+avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](concurrency_control#non-blocking-concurrency-control-mode),
this provides a very powerful way for processing such data streams efficiently and correctly.
### CUSTOM
@@ -249,5 +249,5 @@ Payload class can be specified using the below configs. For more advanced config
There are also quite a few other implementations. Developers may be interested in looking at the hierarchy of `HoodieRecordPayload` interface. For
example, [`MySqlDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/MySqlDebeziumAvroPayload.java) and [`PostgresDebeziumAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/PostgresDebeziumAvroPayload.java) provides support for seamlessly applying changes
captured via Debezium for MySQL and PostgresDB. [`AWSDmsAvroPayload`](https://github.com/apache/hudi/blob/e76dd102bcaf8aec5a932e7277ccdbfd73ce1a32/hudi-common/src/main/java/org/apache/hudi/common/model/AWSDmsAvroPayload.java) provides support for applying changes captured via Amazon Database Migration Service onto S3.
-For full configurations, go [here](/docs/configurations#RECORD_PAYLOAD) and please check out [this FAQ](/docs/next/faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
+For full configurations, go [here](/docs/configurations#RECORD_PAYLOAD) and please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage) if you want to implement your own custom payloads.
diff --git a/website/docs/sql_ddl.md b/website/docs/sql_ddl.md
index e04f64d68b9..c00b815ac4e 100644
--- a/website/docs/sql_ddl.md
+++ b/website/docs/sql_ddl.md
@@ -105,7 +105,7 @@ TBLPROPERTIES (
### Create table with merge modes {#create-table-with-record-merge-mode}
-Hudi supports different [record merge modes](/docs/next/record_merger) to handle merge of incoming records with existing
+Hudi supports different [record merge modes](record_merger) to handle merge of incoming records with existing
records. To create a table with specific record merge mode, you can set `recordMergeMode` option.
```sql
@@ -127,7 +127,7 @@ LOCATION 'file:///tmp/hudi_table_merge_mode/';
With `EVENT_TIME_ORDERING`, the record with the larger event time (`precombineField`) overwrites the record with the
smaller event time on the same key, regardless of transaction's commit time. Users can set `CUSTOM` mode to provide their own
merge logic. With `CUSTOM` merge mode, you can provide a custom class that implements the merge logic. The interfaces
-to implement is explained in detail [here](/docs/next/record_merger#custom).
+to implement is explained in detail [here](record_merger#custom).
```sql
CREATE TABLE IF NOT EXISTS hudi_table_merge_mode_custom (
@@ -236,7 +236,7 @@ AS SELECT * FROM parquet_table;
### Create Index
Hudi supports creating and dropping different types of indexes on a table. For more information on different
-type of indexes please refer [multi-modal indexing](/docs/next/indexes#multi-modal-indexing). Secondary
+type of indexes please refer [multi-modal indexing](indexes#multi-modal-indexing). Secondary
index, expression index and record indexes can be created using SQL create index command.
```sql
@@ -529,7 +529,7 @@ CREATE INDEX idx_bloom_rider ON hudi_indexed_table USING bloom_filters(rider) OP
- Secondary index can only be used for tables using OverwriteWithLatestAvroPayload payload or COMMIT_TIME_ORDERING merge mode
- Column stats Expression Index can not be created using `identity` expression with SQL. Users can leverage column stat index using Datasource instead.
- Index update can fail with schema evolution.
-- Only one index can be created at a time using [async indexer](/docs/next/metadata_indexing).
+- Only one index can be created at a time using [async indexer](metadata_indexing).
### Setting Hudi configs
@@ -592,7 +592,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](/docs/next/concurrency_control#non-blocking-concurrency-control-mode-experimental) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -843,7 +843,7 @@ WITH (
### Create Table in Non-Blocking Concurrency Control Mode
-The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](/docs/next/concurrency_control#non-blocking-concurrency-control).
+The following is an example of creating a Flink table in [Non-Blocking Concurrency Control mode](concurrency_control#non-blocking-concurrency-control).
```sql
-- This is a datagen source that can generate records continuously
@@ -911,7 +911,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](/docs/next/configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/docs/sql_dml.md b/website/docs/sql_dml.md
index 43d5d940fb3..6f5fe28a3eb 100644
--- a/website/docs/sql_dml.md
+++ b/website/docs/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](/docs/next/sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
### Insert Into
@@ -25,7 +25,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](/docs/next/configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -384,7 +384,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](/docs/next/concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/docs/sql_queries.md b/website/docs/sql_queries.md
index b0bda5b6d11..f96ddca3bb6 100644
--- a/website/docs/sql_queries.md
+++ b/website/docs/sql_queries.md
@@ -196,7 +196,7 @@ DROP INDEX partition_stats on hudi_indexed_table;
### Snapshot Query with Event Time Ordering
-Hudi supports different [record merge modes](/docs/next/record_merger) for merging the records from the same key. Event
+Hudi supports different [record merge modes](record_merger) for merging the records from the same key. Event
time ordering is one of the merge modes where the records are merged based on the event time. Let's create a table with
event time ordering merge mode.
diff --git a/website/docs/storage_layouts.md b/website/docs/storage_layouts.md
index 64feb755ba5..a7395ed6e85 100644
--- a/website/docs/storage_layouts.md
+++ b/website/docs/storage_layouts.md
@@ -12,8 +12,8 @@ The following describes the general organization of files in storage for a Hudi
* Each slice contains a **_base file_** (parquet/orc/hfile) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) )
written by a commit that completed at a certain instant, along with set of **_log files_** (*.log.*) written by commits that completed
before the next base file's requested instant.
-* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](/docs/next/compaction) action merges logs and base files to produce new
- file slices and [cleaning](/docs/next/cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+* Hudi employs Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
+ file slices and [cleaning](cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
* All metadata including timeline, metadata table are stored in a special `.hoodie` directory under the base path.
diff --git a/website/docs/use_cases.md b/website/docs/use_cases.md
index 9ad0d255b3d..8120c743607 100644
--- a/website/docs/use_cases.md
+++ b/website/docs/use_cases.md
@@ -65,10 +65,10 @@ together to build out the platform. Such an open platform is also essential for
- While open data formats help, Hudi unlocks complete freedom by also providing open compute services for ingesting, optimizing, indexing and querying data. For e.g Hudi's writers come with
a self-managing table service runtime that can maintain tables automatically in the background on each write. Often times, Hudi and your favorite open query engine is all
you need to get an open data platform up and running.
-- Examples of open services that make performance optimization or management easy include: [auto file sizing](/docs/next/file_sizing) to solve the "small files" problem,
- [clustering](/docs/next/clustering) to co-locate data next to each other, [compaction](/docs/next/compaction) to allow tuning of low latency ingestion + fast read queries,
- [indexing](/docs/next/indexes) - for faster writes/queries, Multi-Dimensional Partitioning (Z-Ordering), automatic cleanup of uncommitted data with marker mechanism,
- [auto cleaning](/docs/next/cleaning) to automatically removing old versions of files.
+- Examples of open services that make performance optimization or management easy include: [auto file sizing](file_sizing) to solve the "small files" problem,
+ [clustering](clustering) to co-locate data next to each other, [compaction](compaction) to allow tuning of low latency ingestion + fast read queries,
+ [indexing](indexes) - for faster writes/queries, Multi-Dimensional Partitioning (Z-Ordering), automatic cleanup of uncommitted data with marker mechanism,
+ [auto cleaning](cleaning) to automatically removing old versions of files.
- Hudi provides rich options for pre-sorting/loading data efficiently and then follow on with rich set of data clustering techniques to manage file sizes and data distribution within a table. In each case, Hudi provides high-degree of configurability in terms of when/how often these services are scheduled, planned and executed. For e.g. Hudi ships with a handful of common planning strategies for compaction and clustering.
- Along with compatibility with other open table formats like [Apache Iceberg](https://iceberg.apache.org/)/[Delta Lake](https://delta.io/), and catalog sync services to various data catalogs, Hudi is one of the most open choices for your data foundation.
diff --git a/website/docs/write_operations.md b/website/docs/write_operations.md
index 0ffff5713e5..d4e8f8fedf2 100644
--- a/website/docs/write_operations.md
+++ b/website/docs/write_operations.md
@@ -120,16 +120,16 @@ Here are the basic configs relevant to the write operations types mentioned abov
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert) : First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/next/indexes) : Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](/docs/next/file_sizing): Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/next/storage_layouts): We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
+2. [Index Lookup](indexes) : Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
+3. [File Sizing](file_sizing): Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
+4. [Partitioning](storage_layouts): We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O :Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/next/indexes): Now that the write is performed, we will go back and update the index.
-7. Commit: Finally we commit all of these changes atomically. ([Post-commit callback](/docs/next/platform_services_post_commit_callback) can be configured.)
-8. [Clean](/docs/next/cleaning) (if needed): Following the commit, cleaning is invoked if needed.
-9. [Compaction](/docs/next/compaction): If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
-10. Archive : Lastly, we perform an archival step which moves old [timeline](/docs/next/timeline) items to an archive folder.
+6. Update [Index](indexes): Now that the write is performed, we will go back and update the index.
+7. Commit: Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback) can be configured.)
+8. [Clean](cleaning) (if needed): Following the commit, cleaning is invoked if needed.
+9. [Compaction](compaction): If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
+10. Archive : Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/docs/writing_data.md b/website/docs/writing_data.md
index 308de8ca78a..81462307a7f 100644
--- a/website/docs/writing_data.md
+++ b/website/docs/writing_data.md
@@ -83,7 +83,7 @@ df.write.format("hudi").
You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<city>/`. We provided a record key
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
-[Modeling data stored in Hudi](/docs/next/faq_general/#how-do-i-model-the-data-stored-in-hudi)
+[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/hoodie_streaming_ingestion).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
@@ -119,7 +119,7 @@ df.write.format("hudi").
You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<city>/`. We provided a record key
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
-[Modeling data stored in Hudi](/docs/next/faq_general/#how-do-i-model-the-data-stored-in-hudi)
+[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/hoodie_streaming_ingestion).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
diff --git a/website/releases/release-0.11.0.md b/website/releases/release-0.11.0.md
index fbea4897b45..dce7ba212fe 100644
--- a/website/releases/release-0.11.0.md
+++ b/website/releases/release-0.11.0.md
@@ -59,7 +59,7 @@ latency with data skipping. Two new indices are added to the metadata table
They are disabled by default. You can enable them by setting `hoodie.metadata.index.bloom.filter.enable`
and `hoodie.metadata.index.column.stats.enable` to `true`, respectively.
-*Refer to the [metadata table guide](/docs/metadata#deployment-considerations) for detailed instructions on upgrade and
+*Refer to the [metadata table guide](/docs/metadata#deployment-considerations-for-metadata-table) for detailed instructions on upgrade and
deployment.*
### Data Skipping with Metadata Table
diff --git a/website/releases/release-0.7.0.md b/website/releases/release-0.7.0.md
index 3573b331093..1eb9274ec68 100644
--- a/website/releases/release-0.7.0.md
+++ b/website/releases/release-0.7.0.md
@@ -64,7 +64,7 @@ Specifically, the `HoodieFlinkStreamer` allows for Hudi Copy-On-Write table to b
derived/ETL pipelines similar to data [sensors](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/sensors/index) in Apache Airflow.
- **Insert Overwrite/Insert Overwrite Table**: We have added these two new write operation types, predominantly to help existing batch ETL jobs, which typically overwrite entire
tables/partitions each run. These operations are much cheaper, than having to issue upserts, given they are bulk replacing the target table.
- Check [here](/docs/quick-start-guide#insert-overwrite-table) for examples.
+ Check [here](/docs/0.7.0/quick-start-guide#insert-overwrite-table) for examples.
- **Delete Partition**: For users of WriteClient/RDD level apis, we have added an API to delete an entire partition, again without issuing deletes at the record level.
- The current default `OverwriteWithLatestAvroPayload` will overwrite the value in storage, even if for e.g the upsert was reissued for an older value of the key.
Added a new `DefaultHoodieRecordPayload` and a new payload config `hoodie.payload.ordering.field` helps specify a field, that the incoming upsert record can be compared with
diff --git a/website/releases/release-0.9.0.md b/website/releases/release-0.9.0.md
index 50b7005e2ff..dbb4b5cc67e 100644
--- a/website/releases/release-0.9.0.md
+++ b/website/releases/release-0.9.0.md
@@ -45,7 +45,7 @@ Hudi tables are now registered with Hive as spark datasource tables, meaning Spa
instead of relying on the Hive fallbacks within Spark, which are ill-maintained/cumbersome. This unlocks many optimizations such as the
use of Hudi's own [FileIndex](https://github.com/apache/hudi/blob/bf5a52e51bbeaa089995335a0a4c55884792e505/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala#L46)
implementation for optimized caching and the use of the Hudi metadata table, for faster listing of large tables. We have also added support for
-[timetravel query](/docs/quick-start-guide#time-travel-query), for spark datasource.
+[timetravel query](/docs/0.9.0/quick-start-guide#time-travel-query), for spark datasource.
### Writer side improvements
diff --git a/website/releases/release-1.0.0-beta1.md b/website/releases/release-1.0.0-beta1.md
index fa8c371b1e3..7fc1b1f6e52 100644
--- a/website/releases/release-1.0.0-beta1.md
+++ b/website/releases/release-1.0.0-beta1.md
@@ -45,7 +45,7 @@ changes in this release:
- Completed actions, their plans and completion metadata are stored in a more
scalable [LSM tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) based timeline organized in an *
*_archived_** storage location under the .hoodie metadata path. It consists of Apache Parquet files with action
- instant data and bookkeeping metadata files, in the following manner. Checkout [timeline](/docs/next/timeline#lsm-timeline) docs for more details.
+ instant data and bookkeeping metadata files, in the following manner. Checkout [timeline](/docs/next/timeline#lsm-timeline-history) docs for more details.
#### Log File Format
@@ -68,7 +68,7 @@ A new concurrency control mode called `NON_BLOCKING_CONCURRENCY_CONTROL` is intr
OCC, multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the
same file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency
mode is currently available for preview in version 1.0.0-beta only. You can read more about it under
-section [Model C: Multi-writer](/docs/next/concurrency_control#non-blocking-concurrency-control-mode-experimental). A complete example with multiple
+section [Model C: Multi-writer](/docs/next/concurrency_control#non-blocking-concurrency-control). A complete example with multiple
Flink streaming writers is available [here](/docs/next/sql_dml#non-blocking-concurrency-control-experimental). You
can follow the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-66/rfc-66.md) and
the [JIRA](https://issues.apache.org/jira/browse/HUDI-6640) for more details.
diff --git a/website/releases/release-1.0.0-beta2.md b/website/releases/release-1.0.0-beta2.md
index bea04c3bfd1..698b2aa3c6e 100644
--- a/website/releases/release-1.0.0-beta2.md
+++ b/website/releases/release-1.0.0-beta2.md
@@ -57,7 +57,7 @@ queries with predicate on columns other than record key columns.
Partition stats index aggregates statistics at the partition level for the columns for which it is enabled. This helps
in efficient partition pruning even for non-partition fields.
-To try out these features, refer to the [SQL guide](/docs/next/sql_ddl#create-partition-stats-and-secondary-index-experimental).
+To try out these features, refer to the [SQL guide](/docs/next/sql_ddl#create-partition-stats-index).
### API Changes
diff --git a/website/versioned_docs/version-0.10.0/clustering.md b/website/versioned_docs/version-0.10.0/clustering.md
index f9bdd572751..e630e92445d 100644
--- a/website/versioned_docs/version-0.10.0/clustering.md
+++ b/website/versioned_docs/version-0.10.0/clustering.md
@@ -12,7 +12,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## Clustering Architecture
-At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations#compactionSmallFileSize) to `0` to force new data [...]
+At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) to `0` to force new [...]
diff --git a/website/versioned_docs/version-0.10.0/compaction.md b/website/versioned_docs/version-0.10.0/compaction.md
index 015d21ec682..e7689b7fdcb 100644
--- a/website/versioned_docs/version-0.10.0/compaction.md
+++ b/website/versioned_docs/version-0.10.0/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.10.0/concurrency_control.md b/website/versioned_docs/version-0.10.0/concurrency_control.md
index 4602ea5df44..54e3f23df4a 100644
--- a/website/versioned_docs/version-0.10.0/concurrency_control.md
+++ b/website/versioned_docs/version-0.10.0/concurrency_control.md
@@ -19,13 +19,13 @@ between multiple table service writers and readers. Additionally, using MVCC, Hu
the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits (or writers) happening to the same table, if they do not have writes to overlapping files being changed, both writers are allowed to succeed.
This feature is currently *experimental* and requires either Zookeeper or HiveMetastore to acquire locks.
-It may be helpful to understand the different guarantees provided by [write operations](/docs/writing_data#write-operations) via Hudi datasource or the delta streamer.
+It may be helpful to understand the different guarantees provided by [write operations](/docs/write_operations) via Hudi datasource or the delta streamer.
## Single Writer Guarantees
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
- - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
- - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order.
## Multi Writer Guarantees
@@ -33,8 +33,8 @@ It may be helpful to understand the different guarantees provided by [write oper
With multiple writers using OCC, some of the above guarantees change as follows
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
-- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
-- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out of order due to multiple writer jobs finishing at different times.
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.10.0/deployment.md b/website/versioned_docs/version-0.10.0/deployment.md
index c3f3de84e88..7614b28c439 100644
--- a/website/versioned_docs/version-0.10.0/deployment.md
+++ b/website/versioned_docs/version-0.10.0/deployment.md
@@ -25,9 +25,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](hoodie_deltastreamer#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -126,7 +126,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.10.0/faq.md b/website/versioned_docs/version-0.10.0/faq.md
index 0af77241155..44d2cabf828 100644
--- a/website/versioned_docs/version-0.10.0/faq.md
+++ b/website/versioned_docs/version-0.10.0/faq.md
@@ -284,7 +284,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.10.0/file_sizing.md b/website/versioned_docs/version-0.10.0/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.10.0/file_sizing.md
+++ b/website/versioned_docs/version-0.10.0/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.10.0/performance.md b/website/versioned_docs/version-0.10.0/performance.md
index 53152730bd8..274ed9dc3fd 100644
--- a/website/versioned_docs/version-0.10.0/performance.md
+++ b/website/versioned_docs/version-0.10.0/performance.md
@@ -14,10 +14,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
-- The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+- The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+- Cleaner can be [configured](/docs/configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations/#hoodielogfilemaxsize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.10.0/querying_data.md b/website/versioned_docs/version-0.10.0/querying_data.md
index 0a4dea31956..3854fb8594b 100644
--- a/website/versioned_docs/version-0.10.0/querying_data.md
+++ b/website/versioned_docs/version-0.10.0/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.10.0/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.10.0/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
@@ -49,7 +49,7 @@ spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hu
```
For examples, refer to [Incremental Queries](/docs/quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+Please refer to [configurations](/docs/configurations/#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
@@ -171,10 +171,10 @@ would ensure Map Reduce execution is chosen for a Hive query, which combines par
separated) and calls InputFormat.listStatus() only once with all those partitions.
## PrestoDB
-To setup PrestoDB for querying Hudi, see the [Query Engine Setup](/docs/0.10.0/query_engine_setup#PrestoDB) page.
+To setup PrestoDB for querying Hudi, see the [Query Engine Setup](/docs/0.10.0/query_engine_setup#prestodb) page.
## Trino
-To setup Trino for querying Hudi, see the [Query Engine Setup](/docs/0.10.0/query_engine_setup#Trino) page.
+To setup Trino for querying Hudi, see the [Query Engine Setup](/docs/0.10.0/query_engine_setup#trino) page.
## Impala (3.4 or later)
diff --git a/website/versioned_docs/version-0.10.0/quick-start-guide.md b/website/versioned_docs/version-0.10.0/quick-start-guide.md
index 281be622248..44b9d38a585 100644
--- a/website/versioned_docs/version-0.10.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.10.0/quick-start-guide.md
@@ -409,7 +409,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
@@ -445,7 +445,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
diff --git a/website/versioned_docs/version-0.10.0/write_operations.md b/website/versioned_docs/version-0.10.0/write_operations.md
index eb3cb9a4522..952fe3b1196 100644
--- a/website/versioned_docs/version-0.10.0/write_operations.md
+++ b/website/versioned_docs/version-0.10.0/write_operations.md
@@ -37,7 +37,7 @@ Hudi supports implementing two types of deletes on data stored in Hudi tables, b
## Writing path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](/docs/configurations/#writeinsertdeduplicate)
+1. [Deduping](configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
2. [Index Lookup](/docs/indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.10.0/writing_data.md b/website/versioned_docs/version-0.10.0/writing_data.md
index 719813360c4..9806ef70648 100644
--- a/website/versioned_docs/version-0.10.0/writing_data.md
+++ b/website/versioned_docs/version-0.10.0/writing_data.md
@@ -93,7 +93,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
@@ -129,7 +129,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
diff --git a/website/versioned_docs/version-0.10.1/clustering.md b/website/versioned_docs/version-0.10.1/clustering.md
index f9bdd572751..e630e92445d 100644
--- a/website/versioned_docs/version-0.10.1/clustering.md
+++ b/website/versioned_docs/version-0.10.1/clustering.md
@@ -12,7 +12,7 @@ Apache Hudi brings stream processing to big data, providing fresh data while bei
## Clustering Architecture
-At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations#compactionSmallFileSize) to `0` to force new data [...]
+At a high level, Hudi provides different operations such as insert/upsert/bulk_insert through it’s write client API to be able to write data to a Hudi table. To be able to choose a trade-off between file size and ingestion speed, Hudi provides a knob `hoodie.parquet.small.file.limit` to be able to configure the smallest allowable file size. Users are able to configure the small file [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) to `0` to force new [...]
diff --git a/website/versioned_docs/version-0.10.1/compaction.md b/website/versioned_docs/version-0.10.1/compaction.md
index c56df32c186..5267f120984 100644
--- a/website/versioned_docs/version-0.10.1/compaction.md
+++ b/website/versioned_docs/version-0.10.1/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.10.1/concurrency_control.md b/website/versioned_docs/version-0.10.1/concurrency_control.md
index 6ea34baa9aa..b926babecb8 100644
--- a/website/versioned_docs/version-0.10.1/concurrency_control.md
+++ b/website/versioned_docs/version-0.10.1/concurrency_control.md
@@ -19,13 +19,13 @@ between multiple table service writers and readers. Additionally, using MVCC, Hu
the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits (or writers) happening to the same table, if they do not have writes to overlapping files being changed, both writers are allowed to succeed.
This feature is currently *experimental* and requires either Zookeeper or HiveMetastore to acquire locks.
-It may be helpful to understand the different guarantees provided by [write operations](/docs/writing_data#write-operations) via Hudi datasource or the delta streamer.
+It may be helpful to understand the different guarantees provided by [write operations](/docs/write_operations) via Hudi datasource or the delta streamer.
## Single Writer Guarantees
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
- - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
- - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order.
## Multi Writer Guarantees
@@ -33,8 +33,8 @@ It may be helpful to understand the different guarantees provided by [write oper
With multiple writers using OCC, some of the above guarantees change as follows
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
-- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
-- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out of order due to multiple writer jobs finishing at different times.
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.10.1/deployment.md b/website/versioned_docs/version-0.10.1/deployment.md
index c3f3de84e88..7614b28c439 100644
--- a/website/versioned_docs/version-0.10.1/deployment.md
+++ b/website/versioned_docs/version-0.10.1/deployment.md
@@ -25,9 +25,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](hoodie_deltastreamer#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -126,7 +126,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.10.1/faq.md b/website/versioned_docs/version-0.10.1/faq.md
index d9691b8bdeb..f6feaa25343 100644
--- a/website/versioned_docs/version-0.10.1/faq.md
+++ b/website/versioned_docs/version-0.10.1/faq.md
@@ -284,7 +284,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.10.1/file_sizing.md b/website/versioned_docs/version-0.10.1/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.10.1/file_sizing.md
+++ b/website/versioned_docs/version-0.10.1/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.10.1/performance.md b/website/versioned_docs/version-0.10.1/performance.md
index 53152730bd8..274ed9dc3fd 100644
--- a/website/versioned_docs/version-0.10.1/performance.md
+++ b/website/versioned_docs/version-0.10.1/performance.md
@@ -14,10 +14,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
-- The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+- The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
-- Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
-- User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+- Cleaner can be [configured](/docs/configurations#hoodiecleanercommitsretained) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+- User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations/#hoodielogfilemaxsize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.10.1/querying_data.md b/website/versioned_docs/version-0.10.1/querying_data.md
index a4fe212de99..4a120b3423f 100644
--- a/website/versioned_docs/version-0.10.1/querying_data.md
+++ b/website/versioned_docs/version-0.10.1/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.10.1/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.10.1/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
@@ -49,7 +49,7 @@ spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hu
```
For examples, refer to [Incremental Queries](/docs/quick-start-guide#incremental-query) in the Spark quickstart.
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+Please refer to [configurations](/docs/configurations/#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
@@ -170,10 +170,10 @@ would ensure Map Reduce execution is chosen for a Hive query, which combines par
separated) and calls InputFormat.listStatus() only once with all those partitions.
## PrestoDB
-To setup PrestoDB for querying Hudi, see the [Query Engine Setup](/docs/0.10.1/query_engine_setup#PrestoDB) page.
+To setup PrestoDB for querying Hudi, see the [Query Engine Setup](/docs/0.10.1/query_engine_setup#prestodb) page.
## Trino
-To setup Trino for querying Hudi, see the [Query Engine Setup](/docs/0.10.1/query_engine_setup#Trino) page.
+To setup Trino for querying Hudi, see the [Query Engine Setup](/docs/0.10.1/query_engine_setup#trino) page.
## Impala (3.4 or later)
diff --git a/website/versioned_docs/version-0.10.1/quick-start-guide.md b/website/versioned_docs/version-0.10.1/quick-start-guide.md
index fc4f17202ff..d286cc3e61c 100644
--- a/website/versioned_docs/version-0.10.1/quick-start-guide.md
+++ b/website/versioned_docs/version-0.10.1/quick-start-guide.md
@@ -426,7 +426,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
@@ -462,7 +462,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
diff --git a/website/versioned_docs/version-0.10.1/tuning-guide.md b/website/versioned_docs/version-0.10.1/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.10.1/tuning-guide.md
+++ b/website/versioned_docs/version-0.10.1/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.10.1/write_operations.md b/website/versioned_docs/version-0.10.1/write_operations.md
index eb3cb9a4522..952fe3b1196 100644
--- a/website/versioned_docs/version-0.10.1/write_operations.md
+++ b/website/versioned_docs/version-0.10.1/write_operations.md
@@ -37,7 +37,7 @@ Hudi supports implementing two types of deletes on data stored in Hudi tables, b
## Writing path
The following is an inside look on the Hudi write path and the sequence of events that occur during a write.
-1. [Deduping](/docs/configurations/#writeinsertdeduplicate)
+1. [Deduping](configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
2. [Index Lookup](/docs/indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.10.1/writing_data.md b/website/versioned_docs/version-0.10.1/writing_data.md
index 719813360c4..9806ef70648 100644
--- a/website/versioned_docs/version-0.10.1/writing_data.md
+++ b/website/versioned_docs/version-0.10.1/writing_data.md
@@ -93,7 +93,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
@@ -129,7 +129,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
:::
</TabItem>
diff --git a/website/versioned_docs/version-0.11.0/compaction.md b/website/versioned_docs/version-0.11.0/compaction.md
index a6249b7ae7c..e99cc2082c5 100644
--- a/website/versioned_docs/version-0.11.0/compaction.md
+++ b/website/versioned_docs/version-0.11.0/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.11.0/deployment.md b/website/versioned_docs/version-0.11.0/deployment.md
index 24ea35e3999..7fbc595b8b2 100644
--- a/website/versioned_docs/version-0.11.0/deployment.md
+++ b/website/versioned_docs/version-0.11.0/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.11.0/faq.md b/website/versioned_docs/version-0.11.0/faq.md
index 6c2c86fef5d..32469d64e81 100644
--- a/website/versioned_docs/version-0.11.0/faq.md
+++ b/website/versioned_docs/version-0.11.0/faq.md
@@ -284,7 +284,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.11.0/file_sizing.md b/website/versioned_docs/version-0.11.0/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.11.0/file_sizing.md
+++ b/website/versioned_docs/version-0.11.0/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.11.0/querying_data.md b/website/versioned_docs/version-0.11.0/querying_data.md
index 6ad05015e75..3a81bc22a17 100644
--- a/website/versioned_docs/version-0.11.0/querying_data.md
+++ b/website/versioned_docs/version-0.11.0/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.11.0/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.11.0/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.11.0/tuning-guide.md b/website/versioned_docs/version-0.11.0/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.11.0/tuning-guide.md
+++ b/website/versioned_docs/version-0.11.0/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.11.0/write_operations.md b/website/versioned_docs/version-0.11.0/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.11.0/write_operations.md
+++ b/website/versioned_docs/version-0.11.0/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.11.1/compaction.md b/website/versioned_docs/version-0.11.1/compaction.md
index 9d73e31bd5b..7b84502c973 100644
--- a/website/versioned_docs/version-0.11.1/compaction.md
+++ b/website/versioned_docs/version-0.11.1/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.11.1/deployment.md b/website/versioned_docs/version-0.11.1/deployment.md
index c8c4e5cefdc..bce07498029 100644
--- a/website/versioned_docs/version-0.11.1/deployment.md
+++ b/website/versioned_docs/version-0.11.1/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.11.1/faq.md b/website/versioned_docs/version-0.11.1/faq.md
index b081d0fe1b0..095480d2984 100644
--- a/website/versioned_docs/version-0.11.1/faq.md
+++ b/website/versioned_docs/version-0.11.1/faq.md
@@ -295,7 +295,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.11.1/file_sizing.md b/website/versioned_docs/version-0.11.1/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.11.1/file_sizing.md
+++ b/website/versioned_docs/version-0.11.1/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.11.1/querying_data.md b/website/versioned_docs/version-0.11.1/querying_data.md
index 4cae617a5b6..b9c2294a83a 100644
--- a/website/versioned_docs/version-0.11.1/querying_data.md
+++ b/website/versioned_docs/version-0.11.1/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.11.1/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.11.1/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.11.1/tuning-guide.md b/website/versioned_docs/version-0.11.1/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.11.1/tuning-guide.md
+++ b/website/versioned_docs/version-0.11.1/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.11.1/write_operations.md b/website/versioned_docs/version-0.11.1/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.11.1/write_operations.md
+++ b/website/versioned_docs/version-0.11.1/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.12.0/compaction.md b/website/versioned_docs/version-0.12.0/compaction.md
index 9d73e31bd5b..7b84502c973 100644
--- a/website/versioned_docs/version-0.12.0/compaction.md
+++ b/website/versioned_docs/version-0.12.0/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.12.0/deployment.md b/website/versioned_docs/version-0.12.0/deployment.md
index 8964d2f9135..1476b051a62 100644
--- a/website/versioned_docs/version-0.12.0/deployment.md
+++ b/website/versioned_docs/version-0.12.0/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.12.0/faq.md b/website/versioned_docs/version-0.12.0/faq.md
index b43043d89cd..cb3225571c2 100644
--- a/website/versioned_docs/version-0.12.0/faq.md
+++ b/website/versioned_docs/version-0.12.0/faq.md
@@ -322,7 +322,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.12.0/file_sizing.md b/website/versioned_docs/version-0.12.0/file_sizing.md
index 0bb0d9b003b..1c1c12fe207 100644
--- a/website/versioned_docs/version-0.12.0/file_sizing.md
+++ b/website/versioned_docs/version-0.12.0/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md
index 4b7e642099f..75bae79386e 100644
--- a/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.0/flink-quick-start-guide.md
@@ -12,7 +12,7 @@ This guide helps you quickly start using Flink on Hudi, and learn different mode
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
- **Configuration** : For [Global Configuration](/docs/0.12.0/flink_configuration#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/0.12.0/flink_configuration#table-options).
- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/0.12.0/hoodie_deltastreamer#cdc-ingestion), [Bulk Insert](/docs/0.12.0/hoodie_deltastreamer#bulk-insert), [Index Bootstrap](/docs/0.12.0/hoodie_deltastreamer#index-bootstrap), [Changelog Mode](/docs/0.12.0/hoodie_deltastreamer#changelog-mode) and [Append Mode](/docs/0.12.0/hoodie_deltastreamer#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/0.12.0/flink_configuration#memory-optimization) and [Write Rate Limit](/docs/0.12.0/flink_configuration#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/0.12.0/query_engine_setup#prestodb).
diff --git a/website/versioned_docs/version-0.12.0/querying_data.md b/website/versioned_docs/version-0.12.0/querying_data.md
index 074d9a2e7c4..70adabf40a6 100644
--- a/website/versioned_docs/version-0.12.0/querying_data.md
+++ b/website/versioned_docs/version-0.12.0/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.0/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.0/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.12.0/tuning-guide.md b/website/versioned_docs/version-0.12.0/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.12.0/tuning-guide.md
+++ b/website/versioned_docs/version-0.12.0/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.12.0/write_operations.md b/website/versioned_docs/version-0.12.0/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.12.0/write_operations.md
+++ b/website/versioned_docs/version-0.12.0/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.12.1/compaction.md b/website/versioned_docs/version-0.12.1/compaction.md
index 9d73e31bd5b..7b84502c973 100644
--- a/website/versioned_docs/version-0.12.1/compaction.md
+++ b/website/versioned_docs/version-0.12.1/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.12.1/deployment.md b/website/versioned_docs/version-0.12.1/deployment.md
index edd7bc69305..4f01c1b3975 100644
--- a/website/versioned_docs/version-0.12.1/deployment.md
+++ b/website/versioned_docs/version-0.12.1/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.12.1/faq.md b/website/versioned_docs/version-0.12.1/faq.md
index 41b76ec6c15..9245d723aa2 100644
--- a/website/versioned_docs/version-0.12.1/faq.md
+++ b/website/versioned_docs/version-0.12.1/faq.md
@@ -322,7 +322,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.12.1/file_sizing.md b/website/versioned_docs/version-0.12.1/file_sizing.md
index 0bb0d9b003b..1c1c12fe207 100644
--- a/website/versioned_docs/version-0.12.1/file_sizing.md
+++ b/website/versioned_docs/version-0.12.1/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.12.1/querying_data.md b/website/versioned_docs/version-0.12.1/querying_data.md
index 332368fcd33..374502e96d2 100644
--- a/website/versioned_docs/version-0.12.1/querying_data.md
+++ b/website/versioned_docs/version-0.12.1/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.1/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.1/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.12.1/tuning-guide.md b/website/versioned_docs/version-0.12.1/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.12.1/tuning-guide.md
+++ b/website/versioned_docs/version-0.12.1/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.12.1/write_operations.md b/website/versioned_docs/version-0.12.1/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.12.1/write_operations.md
+++ b/website/versioned_docs/version-0.12.1/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.12.2/compaction.md b/website/versioned_docs/version-0.12.2/compaction.md
index a6249b7ae7c..e99cc2082c5 100644
--- a/website/versioned_docs/version-0.12.2/compaction.md
+++ b/website/versioned_docs/version-0.12.2/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.12.2/deployment.md b/website/versioned_docs/version-0.12.2/deployment.md
index 18d9259f745..57f1ed35cb4 100644
--- a/website/versioned_docs/version-0.12.2/deployment.md
+++ b/website/versioned_docs/version-0.12.2/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.12.2/faq.md b/website/versioned_docs/version-0.12.2/faq.md
index 2752b49e3a7..0cf53d918d4 100644
--- a/website/versioned_docs/version-0.12.2/faq.md
+++ b/website/versioned_docs/version-0.12.2/faq.md
@@ -342,7 +342,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.12.2/file_sizing.md b/website/versioned_docs/version-0.12.2/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.12.2/file_sizing.md
+++ b/website/versioned_docs/version-0.12.2/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md
index 41fb1dc503a..3d0944ccd2b 100644
--- a/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.2/flink-quick-start-guide.md
@@ -12,7 +12,7 @@ This guide helps you quickly start using Flink on Hudi, and learn different mode
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
- **Configuration** : For [Global Configuration](/docs/0.12.2/flink_configuration#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/0.12.2/flink_configuration#table-options).
- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/0.12.2/hoodie_deltastreamer#cdc-ingestion), [Bulk Insert](/docs/0.12.2/hoodie_deltastreamer#bulk-insert), [Index Bootstrap](/docs/0.12.2/hoodie_deltastreamer#index-bootstrap), [Changelog Mode](/docs/0.12.2/hoodie_deltastreamer#changelog-mode) and [Append Mode](/docs/0.12.2/hoodie_deltastreamer#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/0.12.2/flink_configuration#memory-optimization) and [Write Rate Limit](/docs/0.12.2/flink_configuration#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/0.12.2/query_engine_setup#prestodb).
diff --git a/website/versioned_docs/version-0.12.2/querying_data.md b/website/versioned_docs/version-0.12.2/querying_data.md
index fff64bc0bad..23d1835010a 100644
--- a/website/versioned_docs/version-0.12.2/querying_data.md
+++ b/website/versioned_docs/version-0.12.2/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.2/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.2/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.12.2/quick-start-guide.md b/website/versioned_docs/version-0.12.2/quick-start-guide.md
index 0143f3b9f89..aa108a75a50 100644
--- a/website/versioned_docs/version-0.12.2/quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.2/quick-start-guide.md
@@ -1099,7 +1099,7 @@ For CoW tables, table services work in inline mode by default.
For MoR tables, some async services are enabled by default.
:::note
-Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](/docs/metadata#deployment-considerations) for detailed instructions.
+Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](metadata#deployment-considerations) for detailed instructions.
If you're using Foreach or ForeachBatch streaming sink you must use inline table services, async table services are not supported.
:::
diff --git a/website/versioned_docs/version-0.12.2/tuning-guide.md b/website/versioned_docs/version-0.12.2/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.12.2/tuning-guide.md
+++ b/website/versioned_docs/version-0.12.2/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.12.2/write_operations.md b/website/versioned_docs/version-0.12.2/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.12.2/write_operations.md
+++ b/website/versioned_docs/version-0.12.2/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.12.3/compaction.md b/website/versioned_docs/version-0.12.3/compaction.md
index a6249b7ae7c..e99cc2082c5 100644
--- a/website/versioned_docs/version-0.12.3/compaction.md
+++ b/website/versioned_docs/version-0.12.3/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.12.3/deployment.md b/website/versioned_docs/version-0.12.3/deployment.md
index 998dafa23ed..cd51d9c9cb5 100644
--- a/website/versioned_docs/version-0.12.3/deployment.md
+++ b/website/versioned_docs/version-0.12.3/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.12.3/faq.md b/website/versioned_docs/version-0.12.3/faq.md
index 05b60c270c7..5d5aafa0ed1 100644
--- a/website/versioned_docs/version-0.12.3/faq.md
+++ b/website/versioned_docs/version-0.12.3/faq.md
@@ -342,7 +342,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.12.3/file_sizing.md b/website/versioned_docs/version-0.12.3/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.12.3/file_sizing.md
+++ b/website/versioned_docs/version-0.12.3/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md b/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md
index afffd7f244e..17951814522 100644
--- a/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.3/flink-quick-start-guide.md
@@ -12,7 +12,7 @@ This guide helps you quickly start using Flink on Hudi, and learn different mode
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
- **Configuration** : For [Global Configuration](/docs/0.12.3/flink_configuration#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/0.12.3/flink_configuration#table-options).
- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/0.12.3/hoodie_deltastreamer#cdc-ingestion), [Bulk Insert](/docs/0.12.3/hoodie_deltastreamer#bulk-insert), [Index Bootstrap](/docs/0.12.3/hoodie_deltastreamer#index-bootstrap), [Changelog Mode](/docs/0.12.3/hoodie_deltastreamer#changelog-mode) and [Append Mode](/docs/0.12.3/hoodie_deltastreamer#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/0.12.3/flink_configuration#memory-optimization) and [Write Rate Limit](/docs/0.12.3/flink_configuration#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/0.12.3/query_engine_setup#prestodb).
diff --git a/website/versioned_docs/version-0.12.3/querying_data.md b/website/versioned_docs/version-0.12.3/querying_data.md
index ddd7b5ced13..470fc4f5df9 100644
--- a/website/versioned_docs/version-0.12.3/querying_data.md
+++ b/website/versioned_docs/version-0.12.3/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.3/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.12.3/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.12.3/quick-start-guide.md b/website/versioned_docs/version-0.12.3/quick-start-guide.md
index 3a990aa7443..67418ddd7ce 100644
--- a/website/versioned_docs/version-0.12.3/quick-start-guide.md
+++ b/website/versioned_docs/version-0.12.3/quick-start-guide.md
@@ -1099,7 +1099,7 @@ For CoW tables, table services work in inline mode by default.
For MoR tables, some async services are enabled by default.
:::note
-Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](/docs/metadata#deployment-considerations) for detailed instructions.
+Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](metadata#deployment-considerations) for detailed instructions.
If you're using Foreach or ForeachBatch streaming sink you must use inline table services, async table services are not supported.
:::
diff --git a/website/versioned_docs/version-0.12.3/tuning-guide.md b/website/versioned_docs/version-0.12.3/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.12.3/tuning-guide.md
+++ b/website/versioned_docs/version-0.12.3/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.12.3/write_operations.md b/website/versioned_docs/version-0.12.3/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.12.3/write_operations.md
+++ b/website/versioned_docs/version-0.12.3/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.13.0/compaction.md b/website/versioned_docs/version-0.13.0/compaction.md
index a6249b7ae7c..e99cc2082c5 100644
--- a/website/versioned_docs/version-0.13.0/compaction.md
+++ b/website/versioned_docs/version-0.13.0/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.13.0/deployment.md b/website/versioned_docs/version-0.13.0/deployment.md
index 8ccc654b4f2..2837cc92d43 100644
--- a/website/versioned_docs/version-0.13.0/deployment.md
+++ b/website/versioned_docs/version-0.13.0/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.13.0/faq.md b/website/versioned_docs/version-0.13.0/faq.md
index b0011e89362..6daa604c7a8 100644
--- a/website/versioned_docs/version-0.13.0/faq.md
+++ b/website/versioned_docs/version-0.13.0/faq.md
@@ -342,7 +342,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.13.0/file_sizing.md b/website/versioned_docs/version-0.13.0/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.13.0/file_sizing.md
+++ b/website/versioned_docs/version-0.13.0/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md
index f9f91a4c1e4..8cae9919bc0 100644
--- a/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.13.0/flink-quick-start-guide.md
@@ -12,7 +12,7 @@ This guide helps you quickly start using Flink on Hudi, and learn different mode
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
- **Configuration** : For [Global Configuration](/docs/0.13.0/flink_configuration#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/0.13.0/flink_configuration#table-options).
- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/0.13.0/hoodie_deltastreamer#cdc-ingestion), [Bulk Insert](/docs/0.13.0/hoodie_deltastreamer#bulk-insert), [Index Bootstrap](/docs/0.13.0/hoodie_deltastreamer#index-bootstrap), [Changelog Mode](/docs/0.13.0/hoodie_deltastreamer#changelog-mode) and [Append Mode](/docs/0.13.0/hoodie_deltastreamer#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/0.13.0/flink_configuration#memory-optimization) and [Write Rate Limit](/docs/0.13.0/flink_configuration#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/0.13.0/query_engine_setup#prestodb).
diff --git a/website/versioned_docs/version-0.13.0/querying_data.md b/website/versioned_docs/version-0.13.0/querying_data.md
index b62b5e9f63e..d95f1b4f71f 100644
--- a/website/versioned_docs/version-0.13.0/querying_data.md
+++ b/website/versioned_docs/version-0.13.0/querying_data.md
@@ -17,7 +17,7 @@ In sections, below we will discuss specific setup to access different query type
The Spark Datasource API is a popular way of authoring Spark ETL pipelines. Hudi tables can be queried via the Spark datasource with a simple `spark.read.parquet`.
See the [Spark Quick Start](/docs/quick-start-guide) for more examples of Spark datasource reading queries.
-To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.13.0/query_engine_setup#Spark-DataSource) page.
+To setup Spark for querying Hudi, see the [Query Engine Setup](/docs/0.13.0/query_engine_setup#spark) page.
### Snapshot query {#spark-snap-query}
Retrieve the data table at the present point in time.
diff --git a/website/versioned_docs/version-0.13.0/quick-start-guide.md b/website/versioned_docs/version-0.13.0/quick-start-guide.md
index d4b55283e3f..839747cfbaa 100644
--- a/website/versioned_docs/version-0.13.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.13.0/quick-start-guide.md
@@ -1103,7 +1103,7 @@ For CoW tables, table services work in inline mode by default.
For MoR tables, some async services are enabled by default.
:::note
-Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](/docs/metadata#deployment-considerations) for detailed instructions.
+Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](metadata#deployment-considerations) for detailed instructions.
If you're using Foreach or ForeachBatch streaming sink you must use inline table services, async table services are not supported.
:::
diff --git a/website/versioned_docs/version-0.13.0/tuning-guide.md b/website/versioned_docs/version-0.13.0/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.13.0/tuning-guide.md
+++ b/website/versioned_docs/version-0.13.0/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.13.0/write_operations.md b/website/versioned_docs/version-0.13.0/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.13.0/write_operations.md
+++ b/website/versioned_docs/version-0.13.0/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.13.1/compaction.md b/website/versioned_docs/version-0.13.1/compaction.md
index a6249b7ae7c..e99cc2082c5 100644
--- a/website/versioned_docs/version-0.13.1/compaction.md
+++ b/website/versioned_docs/version-0.13.1/compaction.md
@@ -95,7 +95,7 @@ is enabled by default.
:::
### Hudi Compactor Utility
-Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/deployment#compactions)
+Hudi provides a standalone tool to execute specific compactions asynchronously. Below is an example and you can read more in the [deployment guide](/docs/cli#compactions)
Example:
```properties
diff --git a/website/versioned_docs/version-0.13.1/deployment.md b/website/versioned_docs/version-0.13.1/deployment.md
index 7554cbfa850..3a90bd9bcaa 100644
--- a/website/versioned_docs/version-0.13.1/deployment.md
+++ b/website/versioned_docs/version-0.13.1/deployment.md
@@ -135,7 +135,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.13.1/faq.md b/website/versioned_docs/version-0.13.1/faq.md
index bd6ba91094c..40cdd44df97 100644
--- a/website/versioned_docs/version-0.13.1/faq.md
+++ b/website/versioned_docs/version-0.13.1/faq.md
@@ -342,7 +342,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.13.1/file_sizing.md b/website/versioned_docs/version-0.13.1/file_sizing.md
index e7935445d9e..58831e4b299 100644
--- a/website/versioned_docs/version-0.13.1/file_sizing.md
+++ b/website/versioned_docs/version-0.13.1/file_sizing.md
@@ -21,7 +21,7 @@ and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below whi
be considered a small file. For the initial bootstrap of a Hudi table, tuning record size estimate is also important to
ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average
record size based on previous commit. Hudi will try to add enough records to a small file at write time to get it to the
-configured maximum limit. For e.g , with `compactionSmallFileSize=100MB` and limitFileSize=120MB, Hudi will pick all
+configured maximum limit. For e.g , with `hoodie.parquet.max.file.size=100MB` and hoodie.parquet.small.file.limit=120MB, Hudi will pick all
files < 100MB and try to get them upto 120MB.
### For Merge-On-Read
diff --git a/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md
index 62c5671b1a1..e30598a9e23 100644
--- a/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.13.1/flink-quick-start-guide.md
@@ -12,11 +12,11 @@ This guide helps you quickly start using Flink on Hudi, and learn different mode
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
- **Configuration** : For [Global Configuration](/docs/0.13.1/flink_configuration#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/0.13.1/flink_configuration#table-options).
- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/0.13.1/hoodie_deltastreamer#cdc-ingestion), [Bulk Insert](/docs/0.13.1/hoodie_deltastreamer#bulk-insert), [Index Bootstrap](/docs/0.13.1/hoodie_deltastreamer#index-bootstrap), [Changelog Mode](/docs/0.13.1/hoodie_deltastreamer#changelog-mode) and [Append Mode](/docs/0.13.1/hoodie_deltastreamer#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/0.13.1/flink_configuration#memory-optimization) and [Write Rate Limit](/docs/0.13.1/flink_configuration#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/querying_data#prestodb).
-- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](/docs/querying_data/#hudi-catalog).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](querying_data#prestodb).
+- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](querying_data/#hudi-catalog).
## Quick Start
diff --git a/website/versioned_docs/version-0.13.1/quick-start-guide.md b/website/versioned_docs/version-0.13.1/quick-start-guide.md
index acba2853878..297b05d20bd 100644
--- a/website/versioned_docs/version-0.13.1/quick-start-guide.md
+++ b/website/versioned_docs/version-0.13.1/quick-start-guide.md
@@ -1103,7 +1103,7 @@ For CoW tables, table services work in inline mode by default.
For MoR tables, some async services are enabled by default.
:::note
-Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](/docs/metadata#deployment-considerations) for detailed instructions.
+Since Hudi 0.11 Metadata Table is enabled by default. When using async table services with Metadata Table enabled you must use Optimistic Concurrency Control to avoid the risk of data loss (even in single writer scenario). See [Metadata Table deployment considerations](metadata#deployment-considerations) for detailed instructions.
If you're using Foreach or ForeachBatch streaming sink you must use inline table services, async table services are not supported.
:::
diff --git a/website/versioned_docs/version-0.13.1/record_payload.md b/website/versioned_docs/version-0.13.1/record_payload.md
index 48c3f0e6b79..750e1985863 100644
--- a/website/versioned_docs/version-0.13.1/record_payload.md
+++ b/website/versioned_docs/version-0.13.1/record_payload.md
@@ -139,5 +139,5 @@ Amazon Database Migration Service onto S3.
Record payloads are tunable to suit many use cases. Please check out the configurations
listed [here](/docs/configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
please check
-out [this FAQ](/docs/faq/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
+out [this FAQ](faq/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.13.1/tuning-guide.md b/website/versioned_docs/version-0.13.1/tuning-guide.md
index 4affeafda66..12b68098e06 100644
--- a/website/versioned_docs/version-0.13.1/tuning-guide.md
+++ b/website/versioned_docs/version-0.13.1/tuning-guide.md
@@ -17,7 +17,7 @@ Writing data via Hudi happens as a Spark job and thus general rules of spark deb
**Spark Memory** : Typically, hudi needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some `spark.memory.storageFraction` will generally help boost performance.
-**Sizing files**: Set `limitFileSize` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
+**Sizing files**: Set `hoodie.parquet.small.file.limit` above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.
**Timeseries/Log data** : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases consider tuning the bloom filter accuracy via `.bloomFilterFPP()/bloomFilterNumEntries()` to achieve your target index look up time. Also, consider making a key that is prefixed with time of the event, which will enable r [...]
diff --git a/website/versioned_docs/version-0.13.1/write_operations.md b/website/versioned_docs/version-0.13.1/write_operations.md
index baa6d7dbf84..9ff8431384c 100644
--- a/website/versioned_docs/version-0.13.1/write_operations.md
+++ b/website/versioned_docs/version-0.13.1/write_operations.md
@@ -51,7 +51,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.14.0/deployment.md b/website/versioned_docs/version-0.14.0/deployment.md
index f5ad89c7f81..b400e413a7d 100644
--- a/website/versioned_docs/version-0.14.0/deployment.md
+++ b/website/versioned_docs/version-0.14.0/deployment.md
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.14.0/faq.md b/website/versioned_docs/version-0.14.0/faq.md
index c64c63bda98..74bf66d3ae7 100644
--- a/website/versioned_docs/version-0.14.0/faq.md
+++ b/website/versioned_docs/version-0.14.0/faq.md
@@ -162,7 +162,7 @@ Further - Hudi’s commit time can be a logical time and need not strictly be a
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](https://hudi.apache.org/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](https://hudi.apache.org/docs/hoodie_streaming_ingestion#deltastreamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](https://hudi.apache.org/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](https://hudi.apache.org/docs/0.14.0/hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to captu [...]
### How is a Hudi writer job deployed?
@@ -303,7 +303,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk\_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](https://hudi.apache.org/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](https://hudi.apache.org/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hu [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
diff --git a/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
index 54afa766a19..c64f84a4e60 100644
--- a/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.0/flink-quick-start-guide.md
@@ -11,11 +11,11 @@ This guide helps you quickly start using Flink on Hudi, and learn different mode
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
- **Configuration** : For [Global Configuration](/docs/flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/hoodie_streaming_ingestion#cdc-ingestion), [Bulk Insert](/docs/hoodie_streaming_ingestion#bulk-insert), [Index Bootstrap](/docs/hoodie_streaming_ingestion#index-bootstrap), [Changelog Mode](/docs/hoodie_streaming_ingestion#changelog-mode) and [Append Mode](/docs/hoodie_streaming_ingestion#append-mode).
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](hoodie_streaming_ingestion#cdc-ingestion), [Bulk Insert](hoodie_streaming_ingestion#bulk-insert), [Index Bootstrap](hoodie_streaming_ingestion#index-bootstrap), [Changelog Mode](hoodie_streaming_ingestion#changelog-mode) and [Append Mode](hoodie_streaming_ingestion#append-mode).
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/flink_tuning#memory-optimization) and [Write Rate Limit](/docs/flink_tuning#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/querying_data#prestodb).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](/docs/sql_ddl/#create-catalog).
## Quick Start
diff --git a/website/versioned_docs/version-0.14.0/quick-start-guide.md b/website/versioned_docs/version-0.14.0/quick-start-guide.md
index 1512bab6e05..32c89b2afe0 100644
--- a/website/versioned_docs/version-0.14.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.0/quick-start-guide.md
@@ -1123,9 +1123,9 @@ Hudi provides industry-leading performance and functionality for streaming data.
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](/docs/hoodie_streaming_ingestion#structured-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](hoodie_streaming_ingestion#structured-streaming) for more.
-Check out more information on [modeling data in Hudi](/docs/faq#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](/docs/writing_data).
+Check out more information on [modeling data in Hudi](faq#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](/docs/writing_data).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
diff --git a/website/versioned_docs/version-0.14.0/record_payload.md b/website/versioned_docs/version-0.14.0/record_payload.md
index 1ed47b2ca96..fb63c8f5293 100644
--- a/website/versioned_docs/version-0.14.0/record_payload.md
+++ b/website/versioned_docs/version-0.14.0/record_payload.md
@@ -172,5 +172,5 @@ provides support for applying changes captured via Amazon Database Migration Ser
Record payloads are tunable to suit many use cases. Please check out the configurations
listed [here](/docs/configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
-please check out [this FAQ](/docs/faq/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
+please check out [this FAQ](faq/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.14.0/write_operations.md b/website/versioned_docs/version-0.14.0/write_operations.md
index abd1bdb66db..29132a38f5a 100644
--- a/website/versioned_docs/version-0.14.0/write_operations.md
+++ b/website/versioned_docs/version-0.14.0/write_operations.md
@@ -100,7 +100,7 @@ The following is an inside look on the Hudi write path and the sequence of event
6. Update [Index](/docs/indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/writing_data#commit-notifications) is exposed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
8. [Clean](/docs/hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
9. [Compaction](/docs/compaction)
diff --git a/website/versioned_docs/version-0.14.1/cli.md b/website/versioned_docs/version-0.14.1/cli.md
index 1c30b9b6fa6..7cc4cdd92b0 100644
--- a/website/versioned_docs/version-0.14.1/cli.md
+++ b/website/versioned_docs/version-0.14.1/cli.md
@@ -578,7 +578,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](/docs/next/disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-0.14.1/concurrency_control.md b/website/versioned_docs/version-0.14.1/concurrency_control.md
index dd4e217829e..3efcc049249 100644
--- a/website/versioned_docs/version-0.14.1/concurrency_control.md
+++ b/website/versioned_docs/version-0.14.1/concurrency_control.md
@@ -77,7 +77,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](/docs/next/writing_data#non-blocking-concurrency-control-experimental).
+writers [here](writing_data#non-blocking-concurrency-control-experimental).
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.14.1/deployment.md b/website/versioned_docs/version-0.14.1/deployment.md
index f5ad89c7f81..b400e413a7d 100644
--- a/website/versioned_docs/version-0.14.1/deployment.md
+++ b/website/versioned_docs/version-0.14.1/deployment.md
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inl [...]
+As described in [Writing Data](writing_data#spark-datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.14.1/faq_writing_tables.md b/website/versioned_docs/version-0.14.1/faq_writing_tables.md
index 40c3a99fa99..ae23bbf1e7d 100644
--- a/website/versioned_docs/version-0.14.1/faq_writing_tables.md
+++ b/website/versioned_docs/version-0.14.1/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](/docs/hoodie_streaming_ingestion#deltastreamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasour [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](hoodie_streaming_ingestion#deltastreamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasource API [...]
### How is a Hudi writer job deployed?
@@ -147,7 +147,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk\_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size base [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size base [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
@@ -183,7 +183,7 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](/docs/concurrency_control#supported-concurrency-controls) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-com [...]
+Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
### Can single-writer inserts have duplicates?
diff --git a/website/versioned_docs/version-0.14.1/file_layouts.md b/website/versioned_docs/version-0.14.1/file_layouts.md
index 71ee6d56307..b8b5ca7c342 100644
--- a/website/versioned_docs/version-0.14.1/file_layouts.md
+++ b/website/versioned_docs/version-0.14.1/file_layouts.md
@@ -10,8 +10,8 @@ The following describes the general file layout structure for Apache Hudi. Pleas
* Each file group contains several file slices
* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
-Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/next/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/next/cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
diff --git a/website/versioned_docs/version-0.14.1/file_sizing.md b/website/versioned_docs/version-0.14.1/file_sizing.md
index c637a5a630c..a451b09b6c5 100644
--- a/website/versioned_docs/version-0.14.1/file_sizing.md
+++ b/website/versioned_docs/version-0.14.1/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](/docs/next/cleaning)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md b/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md
index 02e5a19e5f6..74ecc9d73a9 100644
--- a/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.1/flink-quick-start-guide.md
@@ -453,19 +453,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](/docs/next/flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/next/flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/hoodie_streaming_ingestion#cdc-ingestion), [Bulk Insert](/docs/hoodie_streaming_ingestion#bulk-insert), [Index Bootstrap](/docs/hoodie_streaming_ingestion#index-bootstrap), [Changelog Mode](/docs/hoodie_streaming_ingestion#changelog-mode) and [Append Mode](/docs/hoodie_streaming_ingestion#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](/docs/ [...]
-- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/querying_data#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/next/flink_tuning#memory-optimization) and [Write Rate Limit](/docs/next/flink_tuning#write-rate-limit).
+- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](hoodie_streaming_ingestion#cdc-ingestion), [Bulk Insert](hoodie_streaming_ingestion#bulk-insert), [Index Bootstrap](hoodie_streaming_ingestion#index-bootstrap), [Changelog Mode](hoodie_streaming_ingestion#changelog-mode) and [Append Mode](hoodie_streaming_ingestion#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](writing_data#non-blocking-concurrenc [...]
+- **Querying Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](/docs/querying_data#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/querying_data#prestodb).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](/docs/sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](/docs/next/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/next/storage_layouts) - How the files are laid out on storage
- - [Hudi Table Types](/docs/next/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](/docs/next/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline) – How Hudi manages transactions and other table services
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
+ - [Hudi Table Types](table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
diff --git a/website/versioned_docs/version-0.14.1/indexing.md b/website/versioned_docs/version-0.14.1/indexing.md
index 034246ad580..53e883c3856 100644
--- a/website/versioned_docs/version-0.14.1/indexing.md
+++ b/website/versioned_docs/version-0.14.1/indexing.md
@@ -11,9 +11,9 @@ Hudi provides efficient upserts, by mapping a given hoodie key (record key + par
This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
mapped file group contains all versions of a group of records.
-For [Copy-On-Write tables](/docs/next/table_types#copy-on-write-table), this enables fast upsert/delete operations, by
+For [Copy-On-Write tables](table_types#copy-on-write-table), this enables fast upsert/delete operations, by
avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](/docs/next/table_types#merge-on-read-table), this design allows Hudi to bound the amount of
+For [Merge-On-Read tables](table_types#merge-on-read-table), this design allows Hudi to bound the amount of
records any given base file needs to be merged against.
Specifically, a given base file needs to merged only against updates for records that are part of that base file. In contrast,
designs without an indexing component (e.g: [Apache Hive ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)),
diff --git a/website/versioned_docs/version-0.14.1/metadata_indexing.md b/website/versioned_docs/version-0.14.1/metadata_indexing.md
index 5b96ed07bd4..2a0bbfca06f 100644
--- a/website/versioned_docs/version-0.14.1/metadata_indexing.md
+++ b/website/versioned_docs/version-0.14.1/metadata_indexing.md
@@ -78,8 +78,8 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](/docs/next/metadata#metadata-table-indices) along with [configs](/docs/next/metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
-to enable them. The full set of metadata configurations can be explored [here](/docs/next/configurations/#Metadata-Configs).
+metadata table can be explored [here](metadata#metadata-table-indices) along with [configs](metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
+to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
Enabling the metadata table and configuring a lock provider are the prerequisites for using async indexer. Checkout a sample
diff --git a/website/versioned_docs/version-0.14.1/procedures.md b/website/versioned_docs/version-0.14.1/procedures.md
index c2cd0dea7c1..0a895560df0 100644
--- a/website/versioned_docs/version-0.14.1/procedures.md
+++ b/website/versioned_docs/version-0.14.1/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](/docs/next/configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](/docs/next/configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](/docs/next/configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](/docs/next/configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](/docs/next/configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](/docs/next/configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1533,13 +1533,13 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](/docs/next/configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the ma [...]
-| [retain_commits](/docs/next/configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](/docs/next/configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured n [...]
-| [file_versions_retained](/docs/next/configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](/docs/next/configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
-| [trigger_max_commits](/docs/next/configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
-| [options](/docs/next/configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
+| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
+| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
+| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
+| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
**Output**
@@ -1633,12 +1633,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](/docs/next/configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](/docs/next/configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](/docs/next/configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](/docs/next/configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](/docs/next/configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](/docs/next/configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1848,18 +1848,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](/docs/next/configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](/docs/next/configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](/docs/next/configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](/docs/next/configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the nu [...]
+| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-0.14.1/querying_data.md b/website/versioned_docs/version-0.14.1/querying_data.md
index c43ee1fd7f4..ee330fede3a 100644
--- a/website/versioned_docs/version-0.14.1/querying_data.md
+++ b/website/versioned_docs/version-0.14.1/querying_data.md
@@ -7,7 +7,7 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](/docs/next/sql_ddl), [SQL DML](/docs/next/sql_dml), [SQL Queries](/docs/next/sql_queries) and [Procedures](/docs/next/procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
:::
Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](/docs/concepts#query-types).
diff --git a/website/versioned_docs/version-0.14.1/quick-start-guide.md b/website/versioned_docs/version-0.14.1/quick-start-guide.md
index 6f07e7363b6..3c78608036f 100644
--- a/website/versioned_docs/version-0.14.1/quick-start-guide.md
+++ b/website/versioned_docs/version-0.14.1/quick-start-guide.md
@@ -223,7 +223,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](/docs/next/sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
</TabItem>
@@ -267,7 +267,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](/docs/next/write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -300,7 +300,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](/docs/next/write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -309,7 +309,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](/docs/next/sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -421,7 +421,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](/docs/next/sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -451,7 +451,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](/docs/next/concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -539,7 +539,7 @@ MERGE statement either using `SET *` or using `SET column1 = expression1 [, colu
## Delete data {#deletes}
Delete operation removes the records specified from the table. For example, this code snippet deletes records
-for the HoodieKeys passed in. Check out the [deletion section](/docs/next/writing_data#deletes) for more details.
+for the HoodieKeys passed in. Check out the [deletion section](writing_data#deletes) for more details.
<Tabs
groupId="programming-language"
@@ -1035,7 +1035,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](/docs/next/write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1108,29 +1108,29 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](/docs/next/migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](/docs/next/sql_ddl) and [Spark SQL DML](/docs/next/sql_dml) reference guides.
-For alter table commands, check out [this](/docs/next/sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](/docs/next/procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
+For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
### Streaming workloads
Hudi provides industry-leading performance and functionality for streaming data.
-**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](/docs/next/hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
+**Hudi Streamer** - Hudi provides an incremental ingestion/ETL tool - [HoodieStreamer](hoodie_streaming_ingestion#hudi-streamer), to assist with ingesting data into Hudi
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](/docs/next/hoodie_streaming_ingestion#structured-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](hoodie_streaming_ingestion#structured-streaming) for more.
-Check out more information on [modeling data in Hudi](/docs/next/faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](/docs/next/writing_data).
+Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to [writing Hudi Tables](writing_data).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](/docs/next/docker_demo) to get a taste for it.
+steps [here](docker_demo) to get a taste for it.
diff --git a/website/versioned_docs/version-0.14.1/record_payload.md b/website/versioned_docs/version-0.14.1/record_payload.md
index 105a87ae9a0..0f514dced09 100644
--- a/website/versioned_docs/version-0.14.1/record_payload.md
+++ b/website/versioned_docs/version-0.14.1/record_payload.md
@@ -172,6 +172,6 @@ provides support for applying changes captured via Amazon Database Migration Ser
Record payloads are tunable to suit many use cases. Please check out the configurations
listed [here](/docs/configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
-please check out [this FAQ](/docs/next/faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
+please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.14.1/sql_ddl.md b/website/versioned_docs/version-0.14.1/sql_ddl.md
index 0c953905fcb..eb44b5da7c1 100644
--- a/website/versioned_docs/version-0.14.1/sql_ddl.md
+++ b/website/versioned_docs/version-0.14.1/sql_ddl.md
@@ -104,7 +104,7 @@ TBLPROPERTIES (
```
### Create table from an external location
-Often, Hudi tables are created from streaming writers like the [streamer tool](/docs/next/hoodie_streaming_ingestion#hudi-streamer), which
+Often, Hudi tables are created from streaming writers like the [streamer tool](hoodie_streaming_ingestion#hudi-streamer), which
may later need some SQL statements to run on them. You can create an External table using the `location` statement.
```sql
@@ -389,7 +389,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](/docs/next/concurrency_control#non-blocking-concurrency-control-mode-experimental) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -612,7 +612,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](/docs/next/configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-0.14.1/sql_dml.md b/website/versioned_docs/version-0.14.1/sql_dml.md
index 10219843564..fec050936e4 100644
--- a/website/versioned_docs/version-0.14.1/sql_dml.md
+++ b/website/versioned_docs/version-0.14.1/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](/docs/next/sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
### Insert Into
@@ -25,7 +25,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](/docs/next/configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
:::
Examples:
diff --git a/website/versioned_docs/version-0.14.1/use_cases.md b/website/versioned_docs/version-0.14.1/use_cases.md
index edf37a3a4b0..4d06f1e571a 100644
--- a/website/versioned_docs/version-0.14.1/use_cases.md
+++ b/website/versioned_docs/version-0.14.1/use_cases.md
@@ -85,7 +85,7 @@ stream processing world to ensure pipelines don't break from non backwards compa
### ACID Transactions
Along with a table, Apache Hudi brings ACID transactional guarantees to a data lake.
-Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](/docs/next/timeline), stamped with an
+Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline), stamped with an
instant time that denotes the time at which the action
is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between writer processes
(that issue user’s upserts/deletes), table services (that write data/metadata to optimize/perform bookkeeping) and readers
@@ -127,12 +127,12 @@ cost savings for your data lake.
Some examples of the Apache Hudi services that make this performance optimization easy include:
-- [Auto File Sizing](/docs/next/file_sizing) - to solve the "small files" problem.
-- [Clustering](/docs/next/clustering) - to co-locate data next to each other.
-- [Compaction](/docs/next/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/next/indexes) - for efficient upserts and deletes.
+- [Auto File Sizing](file_sizing) - to solve the "small files" problem.
+- [Clustering](clustering) - to co-locate data next to each other.
+- [Compaction](compaction) - to allow tuning of low latency ingestion and fast read queries.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/next/cleaning) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.14.1/write_operations.md b/website/versioned_docs/version-0.14.1/write_operations.md
index d340c590b12..492cd52046c 100644
--- a/website/versioned_docs/version-0.14.1/write_operations.md
+++ b/website/versioned_docs/version-0.14.1/write_operations.md
@@ -88,25 +88,25 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/next/indexes)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](/docs/next/file_sizing)
+3. [File Sizing](file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/next/storage_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/next/indexes)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. (A [callback notification](/docs/next/writing_data#commit-notifications) is exposed)
-8. [Clean](/docs/next/cleaning) (if needed)
+ 1. Finally we commit all of these changes atomically. (A [callback notification](writing_data#commit-notifications) is exposed)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
-9. [Compaction](/docs/next/compaction)
+9. [Compaction](compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
10. Archive
- 1. Lastly, we perform an archival step which moves old [timeline](/docs/next/timeline) items to an archive folder.
+ 1. Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-0.14.1/writing_data.md b/website/versioned_docs/version-0.14.1/writing_data.md
index 10226cf9747..b8613bac3d6 100644
--- a/website/versioned_docs/version-0.14.1/writing_data.md
+++ b/website/versioned_docs/version-0.14.1/writing_data.md
@@ -12,12 +12,12 @@ In this section, we will cover ways to ingest new changes from external sources
Currently Hudi supports following ways to write the data.
- [Hudi Streamer](/docs/hoodie_streaming_ingestion#hudi-streamer)
- [Spark Hudi Datasource](#spark-datasource-writer)
-- [Spark Structured Streaming](/docs/hoodie_streaming_ingestion#structured-streaming)
-- [Spark SQL](/docs/next/sql_ddl#spark-sql)
-- [Flink Writer](/docs/next/hoodie_streaming_ingestion#flink-ingestion)
-- [Flink SQL](/docs/next/sql_ddl#flink)
+- [Spark Structured Streaming](hoodie_streaming_ingestion#structured-streaming)
+- [Spark SQL](sql_ddl#spark-sql)
+- [Flink Writer](hoodie_streaming_ingestion#flink-ingestion)
+- [Flink SQL](sql_ddl#flink)
- [Java Writer](#java-writer)
-- [Kafka Connect](/docs/next/hoodie_streaming_ingestion#kafka-connect-sink)
+- [Kafka Connect](hoodie_streaming_ingestion#kafka-connect-sink)
## Spark Datasource Writer
@@ -98,7 +98,7 @@ df.write.format("hudi").
You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<city>/`. We provided a record key
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
-[Modeling data stored in Hudi](/docs/next/faq_general/#how-do-i-model-the-data-stored-in-hudi)
+[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
@@ -134,7 +134,7 @@ df.write.format("hudi").
You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<city>/`. We provided a record key
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
-[Modeling data stored in Hudi](/docs/next/faq_general/#how-do-i-model-the-data-stored-in-hudi)
+[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
@@ -547,7 +547,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](/docs/next/concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-0.15.0/cli.md b/website/versioned_docs/version-0.15.0/cli.md
index 1c30b9b6fa6..7cc4cdd92b0 100644
--- a/website/versioned_docs/version-0.15.0/cli.md
+++ b/website/versioned_docs/version-0.15.0/cli.md
@@ -578,7 +578,7 @@ Compaction successfully repaired
### Savepoint and Restore
As the name suggest, "savepoint" saves the table as of the commit time, so that it lets you restore the table to this
-savepoint at a later point in time if need be. You can read more about savepoints and restore [here](/docs/next/disaster_recovery)
+savepoint at a later point in time if need be. You can read more about savepoints and restore [here](disaster_recovery)
To trigger savepoint for a hudi table
```java
diff --git a/website/versioned_docs/version-0.15.0/concurrency_control.md b/website/versioned_docs/version-0.15.0/concurrency_control.md
index 64c9af85b66..78724b99165 100644
--- a/website/versioned_docs/version-0.15.0/concurrency_control.md
+++ b/website/versioned_docs/version-0.15.0/concurrency_control.md
@@ -77,7 +77,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](/docs/next/sql_dml#non-blocking-concurrency-control-experimental).
+writers [here](sql_dml#non-blocking-concurrency-control-experimental).
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.15.0/deployment.md b/website/versioned_docs/version-0.15.0/deployment.md
index 9bafde59c46..7785f4ceaca 100644
--- a/website/versioned_docs/version-0.15.0/deployment.md
+++ b/website/versioned_docs/version-0.15.0/deployment.md
@@ -136,7 +136,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Batch Writes](/docs/next/writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.i [...]
+As described in [Batch Writes](writing_data#spark-datasource-api), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.d [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.15.0/faq.md b/website/versioned_docs/version-0.15.0/faq.md
index 1378839b81c..b1284018930 100644
--- a/website/versioned_docs/version-0.15.0/faq.md
+++ b/website/versioned_docs/version-0.15.0/faq.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
The FAQs are split into following pages. Please refer to the specific pages for more info.
-- [General](/docs/next/faq_general)
+- [General](faq_general)
- [Design & Concepts](/docs/next/faq_design_and_concepts)
- [Writing Tables](/docs/next/faq_writing_tables)
- [Reading Tables](/docs/next/faq_reading_tables)
diff --git a/website/versioned_docs/version-0.15.0/faq_general.md b/website/versioned_docs/version-0.15.0/faq_general.md
index 2682d17e950..e9d649e363e 100644
--- a/website/versioned_docs/version-0.15.0/faq_general.md
+++ b/website/versioned_docs/version-0.15.0/faq_general.md
@@ -61,7 +61,7 @@ Nonetheless, Hudi is designed very much like a database and provides similar fun
### How do I model the data stored in Hudi?
-When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](/docs/next/writing_data) for an example.
+When writing data into Hudi, you model the records like how you would on a key-value store - specify a key field (unique for a single partition/across table), a partition field (denotes partition to place key into) and preCombine/combine logic that specifies how to handle duplicates in a batch of records written. This model enables Hudi to enforce primary key constraints like you would get on a database table. See [here](writing_data) for an example.
When querying/reading data, Hudi just presents itself as a json-like hierarchical table, everyone is used to querying using Hive/Spark/Presto over Parquet/Json/Avro.
diff --git a/website/versioned_docs/version-0.15.0/faq_table_services.md b/website/versioned_docs/version-0.15.0/faq_table_services.md
index 0ca730094e4..7ff398687e3 100644
--- a/website/versioned_docs/version-0.15.0/faq_table_services.md
+++ b/website/versioned_docs/version-0.15.0/faq_table_services.md
@@ -50,6 +50,6 @@ Hudi runs cleaner to remove old file versions as part of writing data either in
Yes. Hudi provides the ability to post a callback notification about a write commit. You can use a http hook or choose to
-be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](/docs/next/platform_services_post_commit_callback)
+be notified via a Kafka/pulsar topic or plug in your own implementation to get notified. Please refer [here](platform_services_post_commit_callback)
for details
diff --git a/website/versioned_docs/version-0.15.0/faq_writing_tables.md b/website/versioned_docs/version-0.15.0/faq_writing_tables.md
index f898e0975fa..1edc6b06520 100644
--- a/website/versioned_docs/version-0.15.0/faq_writing_tables.md
+++ b/website/versioned_docs/version-0.15.0/faq_writing_tables.md
@@ -6,7 +6,7 @@ keywords: [hudi, writing, reading]
### What are some ways to write a Hudi table?
-Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](/docs/hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasour [...]
+Typically, you obtain a set of partial updates/inserts from your source and issue [write operations](/docs/write_operations/) against a Hudi table. If you ingesting data from any of the standard sources like Kafka, or tailing DFS, the [delta streamer](/docs/hoodie_streaming_ingestion#hudi-streamer) tool is invaluable and provides an easy, self-managed solution to getting data written into Hudi. You can also write your own code to capture data from a custom source using the Spark datasour [...]
### How is a Hudi writer job deployed?
@@ -68,7 +68,7 @@ As you could see, ([combineAndGetUpdateValue(), getInsertValue()](https://github
### How do I delete records in the dataset using Hudi?
-GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](/docs/next/writing_data#deletes).
+GDPR has made deletes a must-have tool in everyone's data management toolbox. Hudi supports both soft and hard deletes. For details on how to actually perform them, see [here](writing_data#deletes).
### Should I need to worry about deleting all copies of the records in case of duplicates?
@@ -147,7 +147,7 @@ a) **Auto Size small files during ingestion**: This solution trades ingest/writi
Hudi has the ability to maintain a configured target file size, when performing **upsert/insert** operations. (Note: **bulk\_insert** operation does not provide this functionality and is designed as a simpler replacement for normal `spark.write.parquet` )
-For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size base [...]
+For **copy-on-write**, this is as simple as configuring the [maximum size for a base/parquet file](/docs/configurations#hoodieparquetmaxfilesize) and the [soft limit](/docs/configurations#hoodieparquetsmallfilelimit) below which a file should be considered a small file. For the initial bootstrap to Hudi table, tuning record size estimate is also important to ensure sufficient records are bin-packed in a parquet file. For subsequent writes, Hudi automatically uses average record size base [...]
For **merge-on-read**, there are few more configs to set. MergeOnRead works differently for different INDEX choices.
@@ -183,7 +183,7 @@ No, Hudi does not expose uncommitted files/blocks to the readers. Further, Hudi
### How are conflicts detected in Hudi between multiple writers?
-Hudi employs [optimistic concurrency control](/docs/concurrency_control#supported-concurrency-controls) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-com [...]
+Hudi employs [optimistic concurrency control](concurrency_control) between writers, while implementing MVCC based concurrency control between writers and the table services. Concurrent writers to the same table need to be configured with the same lock provider configuration, to safely perform writes. By default (implemented in “[SimpleConcurrentFileWritesConflictResolutionStrategy](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/cli [...]
### Can single-writer inserts have duplicates?
diff --git a/website/versioned_docs/version-0.15.0/file_layouts.md b/website/versioned_docs/version-0.15.0/file_layouts.md
index 478130fbd7e..6fca19dec49 100644
--- a/website/versioned_docs/version-0.15.0/file_layouts.md
+++ b/website/versioned_docs/version-0.15.0/file_layouts.md
@@ -10,8 +10,8 @@ The following describes the general file layout structure for Apache Hudi. Pleas
* Each file group contains several file slices
* Each slice contains a base file (*.parquet/*.orc) (defined by the config - [hoodie.table.base.file.format](https://hudi.apache.org/docs/next/configurations/#hoodietablebasefileformat) ) produced at a certain commit/compaction instant time, along with set of log files (*.log.*) that contain inserts/updates to the base file since the base file was produced.
-Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](/docs/next/compaction) action merges logs and base files to produce new
-file slices and [cleaning](/docs/next/cleaning) action gets rid of unused/older file slices to reclaim space on the file system.
+Hudi adopts Multiversion Concurrency Control (MVCC), where [compaction](compaction) action merges logs and base files to produce new
+file slices and [cleaning](hoodie_cleaner) action gets rid of unused/older file slices to reclaim space on the file system.
diff --git a/website/versioned_docs/version-0.15.0/file_sizing.md b/website/versioned_docs/version-0.15.0/file_sizing.md
index c637a5a630c..a451b09b6c5 100644
--- a/website/versioned_docs/version-0.15.0/file_sizing.md
+++ b/website/versioned_docs/version-0.15.0/file_sizing.md
@@ -148,7 +148,7 @@ while the clustering service runs.
:::note
Hudi always creates immutable files on storage. To be able to do auto-sizing or clustering, Hudi will always create a
-newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](/docs/next/cleaning)
+newer version of the smaller file, resulting in 2 versions of the same file. The [cleaner service](hoodie_cleaner)
will later kick in and delete the older version small file and keep the latest one.
:::
diff --git a/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md b/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md
index c18b0ec1838..cc4cfe9005a 100644
--- a/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md
+++ b/website/versioned_docs/version-0.15.0/flink-quick-start-guide.md
@@ -448,19 +448,19 @@ feature is that it now lets you author streaming pipelines on streaming or batch
## Where To Go From Here?
- **Quick Start** : Read [Quick Start](#quick-start) to get started quickly Flink sql client to write to(read from) Hudi.
-- **Configuration** : For [Global Configuration](/docs/next/flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](/docs/next/flink_tuning#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](/docs/next/ingestion_flink#cdc-ingestion), [Bulk Insert](/docs/next/ingestion_flink#bulk-insert), [Index Bootstrap](/docs/next/ingestion_flink#index-bootstrap), [Changelog Mode](/docs/next/ingestion_flink#changelog-mode) and [Append Mode](/docs/next/ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](/docs/next/sql_dml#non-blocking-conc [...]
-- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](/docs/sql_queries#streaming-query) and [Incremental Query](/docs/sql_queries#incremental-query).
-- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](/docs/next/flink_tuning#memory-optimization) and [Write Rate Limit](/docs/next/flink_tuning#write-rate-limit).
+- **Configuration** : For [Global Configuration](flink_tuning#global-configurations), sets up through `$FLINK_HOME/conf/flink-conf.yaml`. For per job configuration, sets up through [Table Option](flink_tuning#table-options).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC Ingestion](ingestion_flink#cdc-ingestion), [Bulk Insert](ingestion_flink#bulk-insert), [Index Bootstrap](ingestion_flink#index-bootstrap), [Changelog Mode](ingestion_flink#changelog-mode) and [Append Mode](ingestion_flink#append-mode). Flink also supports multiple streaming writers with [non-blocking concurrency control](sql_dml#non-blocking-concurrency-control-experimental).
+- **Reading Data** : Flink supports different modes for reading, such as [Streaming Query](sql_queries#streaming-query) and [Incremental Query](/docs/sql_queries#incremental-query).
+- **Tuning** : For write/read tasks, this guide gives some tuning suggestions, such as [Memory Optimization](flink_tuning#memory-optimization) and [Write Rate Limit](flink_tuning#write-rate-limit).
- **Optimization**: Offline compaction is supported [Offline Compaction](/docs/compaction#flink-offline-compaction).
-- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](/docs/querying_data#prestodb).
+- **Query Engines**: Besides Flink, many other engines are integrated: [Hive Query](/docs/syncing_metastore#flink-setup), [Presto Query](sql_queries#presto).
- **Catalog**: A Hudi specific catalog is supported: [Hudi Catalog](/docs/sql_ddl/#create-catalog).
If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:
- - [Hudi Timeline](/docs/next/timeline) – How Hudi manages transactions and other table services
- - [Hudi File Layout](/docs/next/storage_layouts) - How the files are laid out on storage
- - [Hudi Table Types](/docs/next/table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
- - [Hudi Query Types](/docs/next/table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
+ - [Hudi Timeline](timeline) – How Hudi manages transactions and other table services
+ - [Hudi File Layout](file_layouts) - How the files are laid out on storage
+ - [Hudi Table Types](table_types) – `COPY_ON_WRITE` and `MERGE_ON_READ`
+ - [Hudi Query Types](table_types#query-types) – Snapshot Queries, Incremental Queries, Read-Optimized Queries
See more in the "Concepts" section of the docs.
diff --git a/website/versioned_docs/version-0.15.0/hudi_stack.md b/website/versioned_docs/version-0.15.0/hudi_stack.md
index 203e8ce5947..ab856396db7 100644
--- a/website/versioned_docs/version-0.15.0/hudi_stack.md
+++ b/website/versioned_docs/version-0.15.0/hudi_stack.md
@@ -93,7 +93,7 @@ Hudi provides snapshot isolation for writers and readers, enabling consistent ta
<p align = "center">Figure: Various platform services in Hudi</p>
-Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers. Services, like [Hudi Streamer](./hoodie_streaming_ingestion#hudi-streamer), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (includin [...]
+Platform services offer functionality that is specific to data and workloads, and they sit directly on top of the table services, interfacing with writers and readers. Services, like [Hudi Streamer](./hoodie_streaming_ingestion#hudi-streamer), are specialized in handling data and workloads, seamlessly integrating with Kafka streams and various formats to build data lakes. They support functionalities like automatic checkpoint management, integration with major schema registries (includin [...]
### Query Engines
Apache Hudi is compatible with a wide array of query engines, catering to various analytical needs. For distributed ETL batch processing, Apache Spark is frequently utilized, leveraging its efficient handling of large-scale data. In the realm of streaming use cases, compute engines such as Apache Flink and Apache Spark's Structured Streaming provide robust support when paired with Hudi. Moreover, Hudi supports modern data lake query engines such as Trino and Presto, as well as modern ana [...]
\ No newline at end of file
diff --git a/website/versioned_docs/version-0.15.0/indexing.md b/website/versioned_docs/version-0.15.0/indexing.md
index 009750d9596..4a233123530 100644
--- a/website/versioned_docs/version-0.15.0/indexing.md
+++ b/website/versioned_docs/version-0.15.0/indexing.md
@@ -11,9 +11,9 @@ Hudi provides efficient upserts, by mapping a given hoodie key (record key + par
This mapping between record key and file group/file id, never changes once the first version of a record has been written to a file. In short, the
mapped file group contains all versions of a group of records.
-For [Copy-On-Write tables](/docs/next/table_types#copy-on-write-table), this enables fast upsert/delete operations, by
+For [Copy-On-Write tables](table_types#copy-on-write-table), this enables fast upsert/delete operations, by
avoiding the need to join against the entire dataset to determine which files to rewrite.
-For [Merge-On-Read tables](/docs/next/table_types#merge-on-read-table), this design allows Hudi to bound the amount of
+For [Merge-On-Read tables](table_types#merge-on-read-table), this design allows Hudi to bound the amount of
records any given base file needs to be merged against.
Specifically, a given base file needs to merged only against updates for records that are part of that base file. In contrast,
designs without an indexing component (e.g: [Apache Hive ACID](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions)),
diff --git a/website/versioned_docs/version-0.15.0/metadata_indexing.md b/website/versioned_docs/version-0.15.0/metadata_indexing.md
index 5b96ed07bd4..2a0bbfca06f 100644
--- a/website/versioned_docs/version-0.15.0/metadata_indexing.md
+++ b/website/versioned_docs/version-0.15.0/metadata_indexing.md
@@ -78,8 +78,8 @@ us schedule the indexing for COLUMN_STATS index. First we need to define a prope
As mentioned before, metadata indices are pluggable. One can add any index at any point in time depending on changing
business requirements. Some configurations to enable particular indices are listed below. Currently, available indices under
-metadata table can be explored [here](/docs/next/metadata#metadata-table-indices) along with [configs](/docs/next/metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
-to enable them. The full set of metadata configurations can be explored [here](/docs/next/configurations/#Metadata-Configs).
+metadata table can be explored [here](metadata#metadata-table-indices) along with [configs](metadata#enable-hudi-metadata-table-and-multi-modal-index-in-write-side)
+to enable them. The full set of metadata configurations can be explored [here](configurations/#Metadata-Configs).
:::note
Enabling the metadata table and configuring a lock provider are the prerequisites for using async indexer. Checkout a sample
diff --git a/website/versioned_docs/version-0.15.0/precommit_validator.md b/website/versioned_docs/version-0.15.0/precommit_validator.md
index 5e13fca3dc0..d5faf61057d 100644
--- a/website/versioned_docs/version-0.15.0/precommit_validator.md
+++ b/website/versioned_docs/version-0.15.0/precommit_validator.md
@@ -91,7 +91,7 @@ void validateRecordsBeforeAndAfter(Dataset<Row> before,
```
## Additional Monitoring with Notifications
-Hudi offers a [commit notification service](/docs/next/platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
+Hudi offers a [commit notification service](platform_services_post_commit_callback) that can be configured to trigger notifications about write commits.
The commit notification service can be combined with pre-commit validators to send a notification when a commit fails a validation. This is possible by passing details about the validation as a custom value to the HTTP endpoint.
diff --git a/website/versioned_docs/version-0.15.0/procedures.md b/website/versioned_docs/version-0.15.0/procedures.md
index 4877e7d3001..938ec385d8a 100644
--- a/website/versioned_docs/version-0.15.0/procedures.md
+++ b/website/versioned_docs/version-0.15.0/procedures.md
@@ -472,10 +472,10 @@ archive commits.
|------------------------------------------------------------------------|---------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | N | None | Hudi table name |
| path | String | N | None | Path of table |
-| [min_commits](/docs/next/configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
-| [max_commits](/docs/next/configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
-| [retain_commits](/docs/next/configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
-| [enable_metadata](/docs/next/configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
+| [min_commits](configurations#hoodiekeepmincommits) | Int | N | 20 | Similar to hoodie.keep.max.commits, but controls the minimum number of instants to retain in the active timeline. |
+| [max_commits](configurations#hoodiekeepmaxcommits) | Int | N | 30 | Archiving service moves older entries from timeline into an archived log after each write, to keep the metadata overhead constant, even as the table size grows. This config controls the maximum number of instants to retain in the active timeline. |
+| [retain_commits](configurations#hoodiecommitsarchivalbatch) | Int | N | 10 | Archiving of instants is batched in best-effort manner, to pack more instants into a single archive log. This config controls such archival batch size. |
+| [enable_metadata](configurations#hoodiemetadataenable) | Boolean | N | false | Enable the internal metadata table |
**Output**
@@ -672,7 +672,7 @@ copy table to a temporary view.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|---------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](/docs/next/configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| view_name | String | Y | None | Name of view |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -705,7 +705,7 @@ copy table to a new table.
| Parameter Name | Type | Required | Default Value | Description |
|-------------------------------------------------------------------|--------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| table | String | Y | None | Hudi table name |
-| [query_type](/docs/next/configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
+| [query_type](configurations#hoodiedatasourcequerytype) | String | N | "snapshot" | Whether data needs to be read, in `incremental` mode (new data since an instantTime) (or) `read_optimized` mode (obtain latest view, based on base files) (or) `snapshot` mode (obtain latest view, by merging base and (if any) log files) |
| new_table | String | Y | None | Name of new table |
| begin_instance_time | String | N | "" | Begin instance time |
| end_instance_time | String | N | "" | End instance time |
@@ -1533,13 +1533,13 @@ Run cleaner on a hoodie table.
|---------------------------------------------------------------------------------------|---------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...]
| table | String | Y | None | Name of table to be cleaned [...]
| schedule_in_line | Boolean | N | true | Set "true" if you want to schedule and run a clean. Set false if you have already scheduled a clean and want to run that. [...]
-| [clean_policy](/docs/next/configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the ma [...]
-| [retain_commits](/docs/next/configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
-| [hours_retained](/docs/next/configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured n [...]
-| [file_versions_retained](/docs/next/configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
-| [trigger_strategy](/docs/next/configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
-| [trigger_max_commits](/docs/next/configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
-| [options](/docs/next/configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
+| [clean_policy](configurations#hoodiecleanerpolicy) | String | N | None | org.apache.hudi.common.model.HoodieCleaningPolicy: Cleaning policy to be used. The cleaner service deletes older file slices files to re-claim space. Long running query plans may often refer to older file slices and will break if those are cleaned, before the query has had a chance to run. So, it is good to make sure that the data is retained for more than the maximum query [...]
+| [retain_commits](configurations#hoodiecleanercommitsretained) | Int | N | None | When KEEP_LATEST_COMMITS cleaning policy is used, the number of commits to retain, without cleaning. This will be retained for num_of_commits * time_between_commits (scheduled). This also directly translates into how much data retention the table supports for incremental queries. [...]
+| [hours_retained](configurations#hoodiecleanerhoursretained) | Int | N | None | When KEEP_LATEST_BY_HOURS cleaning policy is used, the number of hours for which commits need to be retained. This config provides a more flexible option as compared to number of commits retained for cleaning service. Setting this property ensures all the files, but the latest in a file group, corresponding to commits with commit times older than the configured number of ho [...]
+| [file_versions_retained](configurations#hoodiecleanerfileversionsretained) | Int | N | None | When KEEP_LATEST_FILE_VERSIONS cleaning policy is used, the minimum number of file slices to retain in each file group, during cleaning. [...]
+| [trigger_strategy](configurations#hoodiecleantriggerstrategy) | String | N | None | org.apache.hudi.table.action.clean.CleaningTriggerStrategy: Controls when cleaning is scheduled. NUM_COMMITS(default): Trigger the cleaning service every N commits, determined by `hoodie.clean.max.commits` [...]
+| [trigger_max_commits](configurations/#hoodiecleanmaxcommits) | Int | N | None | Number of commits after the last clean operation, before scheduling of a new clean is attempted. [...]
+| [options](configurations/#Clean-Configs) | String | N | None | comma separated list of Hudi configs for cleaning in the format "config1=value1,config2=value2" [...]
**Output**
@@ -1633,12 +1633,12 @@ Sync the table's latest schema to Hive metastore.
| metastore_uri | String | N | "" | Metastore_uri |
| username | String | N | "" | User name |
| password | String | N | "" | Password |
-| [use_jdbc](/docs/next/configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
-| [mode](/docs/next/configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
-| [partition_fields](/docs/next/configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
-| [partition_extractor_class](/docs/next/configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
-| [strategy](/docs/next/configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
-| [sync_incremental](/docs/next/configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
+| [use_jdbc](configurations#hoodiedatasourcehive_syncuse_jdbc) | String | N | "" | Use JDBC when hive synchronization is enabled |
+| [mode](configurations#hoodiedatasourcehive_syncmode) | String | N | "" | Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql. |
+| [partition_fields](configurations#hoodiedatasourcehive_syncpartition_fields) | String | N | "" | Field in the table to use for determining hive partition columns. | |
+| [partition_extractor_class](configurations#hoodiedatasourcehive_syncpartition_extractor_class) | String | N | "" | Class which implements PartitionValueExtractor to extract the partition values, default 'org.apache.hudi.hive.MultiPartKeysValueExtractor'. |
+| [strategy](configurations#hoodiedatasourcehive_synctablestrategy) | String | N | "" | Hive table synchronization strategy. Available option: RO, RT, ALL. |
+| [sync_incremental](configurations#hoodiemetasyncincremental) | String | N | "" | Whether to incrementally sync the partitions to the metastore, i.e., only added, changed, and deleted partitions based on the commit metadata. If set to `false`, the meta sync executes a full partition sync operation when partitions are lost. |
@@ -1848,18 +1848,18 @@ Convert an existing table to Hudi.
|------------------------------------------------------------------------------|---------|----------|-------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
| table | String | Y | None | Name of table to be clustered [...]
| table_type | String | Y | None | Table type, MERGE_ON_READ or COPY_ON_WRITE [...]
-| [bootstrap_path](/docs/next/configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
+| [bootstrap_path](configurations#hoodiebootstrapbasepath) | String | Y | None | Base path of the dataset that needs to be bootstrapped as a Hudi table [...]
| base_path | String | Y | None | Base path [...]
| rowKey_field | String | Y | None | Primary key field [...]
| base_file_format | String | N | "PARQUET" | Format of base file [...]
| partition_path_field | String | N | "" | Partitioned column field [...]
-| [bootstrap_index_class](/docs/next/configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
-| [selector_class](/docs/next/configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
+| [bootstrap_index_class](configurations#hoodiebootstrapindexclass) | String | N | "org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex" | Implementation to use, for mapping a skeleton base file to a bootstrap base file. [...]
+| [selector_class](configurations#hoodiebootstrapmodeselector) | String | N | "org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector" | Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped [...]
| key_generator_class | String | N | "org.apache.hudi.keygen.SimpleKeyGenerator" | Class of key generator [...]
| full_bootstrap_input_provider | String | N | "org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider" | Class of full bootstrap input provider [...]
| schema_provider_class | String | N | "" | Class of schema provider [...]
| payload_class | String | N | "org.apache.hudi.common.model.OverwriteWithLatestAvroPayload" | Class of payload [...]
-| [parallelism](/docs/next/configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the nu [...]
+| [parallelism](configurations#hoodiebootstrapparallelism) | Int | N | 1500 | For metadata-only bootstrap, Hudi parallelizes the operation so that each table partition is handled by one Spark task. This config limits the number of parallelism. We pick the configured parallelism if the number of table partitions is larger than this configured value. The parallelism is assigned to the number of tab [...]
| enable_hive_sync | Boolean | N | false | Whether to enable hive sync [...]
| props_file_path | String | N | "" | Path of properties file [...]
| bootstrap_overwrite | Boolean | N | false | Overwrite bootstrap path [...]
diff --git a/website/versioned_docs/version-0.15.0/querying_data.md b/website/versioned_docs/version-0.15.0/querying_data.md
index 31069822df7..83a03a4a112 100644
--- a/website/versioned_docs/version-0.15.0/querying_data.md
+++ b/website/versioned_docs/version-0.15.0/querying_data.md
@@ -7,7 +7,7 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
:::danger
-This page is no longer maintained. Please refer to Hudi [SQL DDL](/docs/next/sql_ddl), [SQL DML](/docs/next/sql_dml), [SQL Queries](/docs/next/sql_queries) and [Procedures](/docs/next/procedures) for the latest documentation.
+This page is no longer maintained. Please refer to Hudi [SQL DDL](sql_ddl), [SQL DML](sql_dml), [SQL Queries](sql_queries) and [Procedures](procedures) for the latest documentation.
:::
Conceptually, Hudi stores data physically once on DFS, while providing 3 different ways of querying, as explained [before](/docs/concepts#query-types).
diff --git a/website/versioned_docs/version-0.15.0/quick-start-guide.md b/website/versioned_docs/version-0.15.0/quick-start-guide.md
index 0ff3225da1c..b100fb9a85e 100644
--- a/website/versioned_docs/version-0.15.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.15.0/quick-start-guide.md
@@ -255,7 +255,7 @@ CREATE TABLE hudi_table (
PARTITIONED BY (city);
```
-For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](/docs/next/sql_ddl) reference guide.
+For more options for creating Hudi tables or if you're running into any issues, please refer to [SQL DDL](sql_ddl) reference guide.
</TabItem>
@@ -299,7 +299,7 @@ inserts.write.format("hudi").
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](/docs/next/write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -332,7 +332,7 @@ inserts.write.format("hudi"). \
```
:::info Mapping to Hudi write operations
-Hudi provides a wide range of [write operations](/docs/next/write_operations) - both batch and incremental - to write data into Hudi tables,
+Hudi provides a wide range of [write operations](write_operations) - both batch and incremental - to write data into Hudi tables,
with different semantics and performance. When record keys are not configured (see [keys](#keys) below), `bulk_insert` will be chosen as
the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
:::
@@ -341,7 +341,7 @@ the write operation, matching the out-of-behavior of Spark's Parquet Datasource.
<TabItem value="sparksql">
-Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](/docs/next/sql_dml#insert-into) for more advanced options.
+Users can use 'INSERT INTO' to insert data into a Hudi table. See [Insert Into](sql_dml#insert-into) for more advanced options.
```sql
INSERT INTO hudi_table
@@ -453,7 +453,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</TabItem>
<TabItem value="sparksql">
-Hudi table can be update using a regular UPDATE statement. See [Update](/docs/next/sql_dml#update) for more advanced options.
+Hudi table can be update using a regular UPDATE statement. See [Update](sql_dml#update) for more advanced options.
```sql
UPDATE hudi_table SET fare = 25.0 WHERE rider = 'rider-D';
@@ -483,7 +483,7 @@ Notice that the save mode is now `Append`. In general, always use append mode un
</Tabs
>
-[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](/docs/next/concepts).
+[Querying](#querying) the data again will now show updated records. Each write operation generates a new [commit](concepts).
Look for changes in `_hoodie_commit_time`, `fare` fields for the given `_hoodie_record_key` value from a previous commit.
## Merging Data {#merge}
@@ -1067,7 +1067,7 @@ PARTITIONED BY (city);
>
:::note Implications of defining record keys
-Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](/docs/next/write_operations).
+Configuring keys for a Hudi table, has a new implications on the table. If record key is set by the user, `upsert` is chosen as the [write operation](write_operations).
Also if a record key is configured, then it's also advisable to specify a precombine or ordering field, to correctly handle cases where the source data has
multiple records with the same key. See section below.
:::
@@ -1140,12 +1140,12 @@ PARTITIONED BY (city);
## Where to go from here?
You can also [build hudi yourself](https://github.com/apache/hudi#building-apache-hudi-from-source) and try this quickstart using `--jars <path to spark bundle jar>`(see also [build with scala 2.12](https://github.com/apache/hudi#build-with-different-spark-versions))
-for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](/docs/next/migration_guide).
+for more info. If you are looking for ways to migrate your existing data to Hudi, refer to [migration guide](migration_guide).
### Spark SQL Reference
-For advanced usage of spark SQL, please refer to [Spark SQL DDL](/docs/next/sql_ddl) and [Spark SQL DML](/docs/next/sql_dml) reference guides.
-For alter table commands, check out [this](/docs/next/sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](/docs/next/procedures) out.
+For advanced usage of spark SQL, please refer to [Spark SQL DDL](sql_ddl) and [Spark SQL DML](sql_dml) reference guides.
+For alter table commands, check out [this](sql_ddl#spark-alter-table). Stored procedures provide a lot of powerful capabilities using Hudi SparkSQL to assist with monitoring, managing and operating Hudi tables, please check [this](procedures) out.
### Streaming workloads
@@ -1155,14 +1155,14 @@ Hudi provides industry-leading performance and functionality for streaming data.
from various different sources in a streaming manner, with powerful built-in capabilities like auto checkpointing, schema enforcement via schema provider,
transformation support, automatic table services and so on.
-**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](/docs/next/writing_tables_streaming_writes#spark-streaming) for more.
+**Structured Streaming** - Hudi supports Spark Structured Streaming reads and writes as well. Please see [here](writing_tables_streaming_writes#spark-streaming) for more.
-Check out more information on [modeling data in Hudi](/docs/next/faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](/docs/writing_data) and [streaming writes](/docs/next/writing_tables_streaming_writes).
+Check out more information on [modeling data in Hudi](faq_general#how-do-i-model-the-data-stored-in-hudi) and different ways to perform [batch writes](/docs/writing_data) and [streaming writes](writing_tables_streaming_writes).
### Dockerized Demo
Even as we showcased the core capabilities, Hudi supports a lot more advanced functionality that can make it easy
to get your transactional data lakes up and running quickly, across a variety query engines like Hive, Flink, Spark, Presto, Trino and much more.
We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases all of this on a docker based setup with all
dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, by following
-steps [here](/docs/next/docker_demo) to get a taste for it.
+steps [here](docker_demo) to get a taste for it.
diff --git a/website/versioned_docs/version-0.15.0/record_payload.md b/website/versioned_docs/version-0.15.0/record_payload.md
index 105a87ae9a0..0f514dced09 100644
--- a/website/versioned_docs/version-0.15.0/record_payload.md
+++ b/website/versioned_docs/version-0.15.0/record_payload.md
@@ -172,6 +172,6 @@ provides support for applying changes captured via Amazon Database Migration Ser
Record payloads are tunable to suit many use cases. Please check out the configurations
listed [here](/docs/configurations#RECORD_PAYLOAD). Moreover, if users want to implement their own custom merge logic,
-please check out [this FAQ](/docs/next/faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
+please check out [this FAQ](faq_writing_tables/#can-i-implement-my-own-logic-for-how-input-records-are-merged-with-record-on-storage). In a
separate document, we will talk about a new record merger API for optimized payload handling.
diff --git a/website/versioned_docs/version-0.15.0/sql_ddl.md b/website/versioned_docs/version-0.15.0/sql_ddl.md
index 4651e819cdb..56dd6a67756 100644
--- a/website/versioned_docs/version-0.15.0/sql_ddl.md
+++ b/website/versioned_docs/version-0.15.0/sql_ddl.md
@@ -389,7 +389,7 @@ Users can set table properties while creating a table. The important table prope
#### Passing Lock Providers for Concurrent Writers
Hudi requires a lock provider to support concurrent writers or asynchronous table services when using OCC
-and [NBCC](/docs/next/concurrency_control#non-blocking-concurrency-control-mode-experimental) (Non-Blocking Concurrency Control)
+and [NBCC](concurrency_control#non-blocking-concurrency-control) (Non-Blocking Concurrency Control)
concurrency mode. For NBCC mode, locking is only used to write the commit metadata file in the timeline. Writes are
serialized by completion time. Users can pass these table properties into *TBLPROPERTIES* as well. Below is an example
for a Zookeeper based configuration.
@@ -612,7 +612,7 @@ ALTER TABLE tableA RENAME TO tableB;
### Setting Hudi configs
#### Using table options
-You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](/docs/next/configurations#FLINK_SQL)
+You can configure hoodie configs in table options when creating a table. You can refer Flink specific hoodie configs [here](configurations#FLINK_SQL)
These configs will be applied to all the operations on that table.
```sql
diff --git a/website/versioned_docs/version-0.15.0/sql_dml.md b/website/versioned_docs/version-0.15.0/sql_dml.md
index edb63730b13..b94b382df68 100644
--- a/website/versioned_docs/version-0.15.0/sql_dml.md
+++ b/website/versioned_docs/version-0.15.0/sql_dml.md
@@ -12,7 +12,7 @@ import TabItem from '@theme/TabItem';
SparkSQL provides several Data Manipulation Language (DML) actions for interacting with Hudi tables. These operations allow you to insert, update, merge and delete data
from your Hudi tables. Let's explore them one by one.
-Please refer to [SQL DDL](/docs/next/sql_ddl) for creating Hudi tables using SQL.
+Please refer to [SQL DDL](sql_ddl) for creating Hudi tables using SQL.
### Insert Into
@@ -25,7 +25,7 @@ SELECT <columns> FROM <source>;
:::note Deprecations
From 0.14.0, `hoodie.sql.bulk.insert.enable` and `hoodie.sql.insert.mode` are deprecated. Users are expected to use `hoodie.spark.sql.insert.into.operation` instead.
-To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](/docs/next/configurations#hoodiedatasourceinsertduppolicy).
+To manage duplicates with `INSERT INTO`, please check out [insert dup policy config](configurations#hoodiedatasourceinsertduppolicy).
:::
Examples:
@@ -317,7 +317,7 @@ INSERT INTO hudi_table select ... from ...;
Hudi Flink supports a new non-blocking concurrency control mode, where multiple writer tasks can be executed
concurrently without blocking each other. One can read more about this mode in
-the [concurrency control](/docs/next/concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
+the [concurrency control](concurrency_control#model-c-multi-writer) docs. Let us see it in action here.
In the below example, we have two streaming ingestion pipelines that concurrently update the same table. One of the
pipeline is responsible for the compaction and cleaning table services, while the other pipeline is just for data
diff --git a/website/versioned_docs/version-0.15.0/use_cases.md b/website/versioned_docs/version-0.15.0/use_cases.md
index edf37a3a4b0..4d06f1e571a 100644
--- a/website/versioned_docs/version-0.15.0/use_cases.md
+++ b/website/versioned_docs/version-0.15.0/use_cases.md
@@ -85,7 +85,7 @@ stream processing world to ensure pipelines don't break from non backwards compa
### ACID Transactions
Along with a table, Apache Hudi brings ACID transactional guarantees to a data lake.
-Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](/docs/next/timeline), stamped with an
+Hudi ensures atomic writes, by way of publishing commits atomically to a [timeline](timeline), stamped with an
instant time that denotes the time at which the action
is deemed to have occurred. Unlike general purpose file version control, Hudi draws clear distinction between writer processes
(that issue user’s upserts/deletes), table services (that write data/metadata to optimize/perform bookkeeping) and readers
@@ -127,12 +127,12 @@ cost savings for your data lake.
Some examples of the Apache Hudi services that make this performance optimization easy include:
-- [Auto File Sizing](/docs/next/file_sizing) - to solve the "small files" problem.
-- [Clustering](/docs/next/clustering) - to co-locate data next to each other.
-- [Compaction](/docs/next/compaction) - to allow tuning of low latency ingestion and fast read queries.
-- [Indexing](/docs/next/indexes) - for efficient upserts and deletes.
+- [Auto File Sizing](file_sizing) - to solve the "small files" problem.
+- [Clustering](clustering) - to co-locate data next to each other.
+- [Compaction](compaction) - to allow tuning of low latency ingestion and fast read queries.
+- [Indexing](indexing) - for efficient upserts and deletes.
- Multi-Dimensional Partitioning (Z-Ordering) - Traditional folder style partitioning on low-cardinality, while also
Z-Ordering data within files based on high-cardinality
- Metadata Table - No more slow S3 file listings or throttling.
-- [Auto Cleaning](/docs/next/cleaning) - Keeps your storage costs in check by automatically removing old versions of files.
+- [Auto Cleaning](hoodie_cleaner) - Keeps your storage costs in check by automatically removing old versions of files.
diff --git a/website/versioned_docs/version-0.15.0/write_operations.md b/website/versioned_docs/version-0.15.0/write_operations.md
index df0a22e01f9..3d48a9f4ebe 100644
--- a/website/versioned_docs/version-0.15.0/write_operations.md
+++ b/website/versioned_docs/version-0.15.0/write_operations.md
@@ -87,25 +87,25 @@ The following is an inside look on the Hudi write path and the sequence of event
1. [Deduping](/docs/configurations#hoodiecombinebeforeinsert)
1. First your input records may have duplicate keys within the same batch and duplicates need to be combined or reduced by key.
-2. [Index Lookup](/docs/next/indexes)
+2. [Index Lookup](indexing)
1. Next, an index lookup is performed to try and match the input records to identify which file groups they belong to.
-3. [File Sizing](/docs/next/file_sizing)
+3. [File Sizing](file_sizing)
1. Then, based on the average size of previous commits, Hudi will make a plan to add enough records to a small file to get it close to the configured maximum limit.
-4. [Partitioning](/docs/next/storage_layouts)
+4. [Partitioning](file_layouts)
1. We now arrive at partitioning where we decide what file groups certain updates and inserts will be placed in or if new file groups will be created
5. Write I/O
1. Now we actually do the write operations which is either creating a new base file, appending to the log file,
or versioning an existing base file.
-6. Update [Index](/docs/next/indexes)
+6. Update [Index](indexing)
1. Now that the write is performed, we will go back and update the index.
7. Commit
- 1. Finally we commit all of these changes atomically. ([Post-commit callback](/docs/next/platform_services_post_commit_callback) can be configured.)
-8. [Clean](/docs/next/cleaning) (if needed)
+ 1. Finally we commit all of these changes atomically. ([Post-commit callback](platform_services_post_commit_callback) can be configured.)
+8. [Clean](hoodie_cleaner) (if needed)
1. Following the commit, cleaning is invoked if needed.
-9. [Compaction](/docs/next/compaction)
+9. [Compaction](compaction)
1. If you are using MOR tables, compaction will either run inline, or be scheduled asynchronously
10. Archive
- 1. Lastly, we perform an archival step which moves old [timeline](/docs/next/timeline) items to an archive folder.
+ 1. Lastly, we perform an archival step which moves old [timeline](timeline) items to an archive folder.
Here is a diagramatic representation of the flow.
diff --git a/website/versioned_docs/version-0.15.0/writing_data.md b/website/versioned_docs/version-0.15.0/writing_data.md
index fd5e6582ad3..2630cc299db 100644
--- a/website/versioned_docs/version-0.15.0/writing_data.md
+++ b/website/versioned_docs/version-0.15.0/writing_data.md
@@ -83,7 +83,7 @@ df.write.format("hudi").
You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<city>/`. We provided a record key
(`uuid` in [schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/6f9b02decb5bb2b83709b1b6ec04a97e4d102c11/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
-[Modeling data stored in Hudi](/docs/next/faq_general/#how-do-i-model-the-data-stored-in-hudi)
+[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/hoodie_streaming_ingestion).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
@@ -119,7 +119,7 @@ df.write.format("hudi").
You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<city>/`. We provided a record key
(`uuid` in [schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)), partition field (`region/country/city`) and combine logic (`ts` in
[schema](https://github.com/apache/hudi/blob/2e6e302efec2fa848ded4f88a95540ad2adb7798/hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/QuickstartUtils.java#L60)) to ensure trip records are unique within each partition. For more info, refer to
-[Modeling data stored in Hudi](/docs/next/faq_general/#how-do-i-model-the-data-stored-in-hudi)
+[Modeling data stored in Hudi](faq_general/#how-do-i-model-the-data-stored-in-hudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/hoodie_streaming_ingestion).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/write_operations)
diff --git a/website/versioned_docs/version-0.5.0/querying_data.md b/website/versioned_docs/version-0.5.0/querying_data.md
index 49f6dd4cc30..1985ab3d2f7 100644
--- a/website/versioned_docs/version-0.5.0/querying_data.md
+++ b/website/versioned_docs/version-0.5.0/querying_data.md
@@ -129,7 +129,7 @@ A sample incremental pull, that will obtain all records written since `beginInst
.load(tablePath); // For incremental view, pass in the root/base path of dataset
```
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.5.0/quick-start-guide.md b/website/versioned_docs/version-0.5.0/quick-start-guide.md
index 92660aea936..7318eae83d3 100644
--- a/website/versioned_docs/version-0.5.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.5.0/quick-start-guide.md
@@ -64,7 +64,7 @@ You can check the data generated under `/tmp/hudi_cow_table/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Datasets](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
diff --git a/website/versioned_docs/version-0.5.0/writing_data.md b/website/versioned_docs/version-0.5.0/writing_data.md
index 7510c3408dc..61cf8b83ba0 100644
--- a/website/versioned_docs/version-0.5.0/writing_data.md
+++ b/website/versioned_docs/version-0.5.0/writing_data.md
@@ -212,10 +212,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi datasets.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.5.1/deployment.md b/website/versioned_docs/version-0.5.1/deployment.md
index 72ce29c8697..3bda49ed631 100644
--- a/website/versioned_docs/version-0.5.1/deployment.md
+++ b/website/versioned_docs/version-0.5.1/deployment.md
@@ -29,9 +29,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -130,7 +130,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.5.1/querying_data.md b/website/versioned_docs/version-0.5.1/querying_data.md
index 1f1cba44c1a..978fbdc60f9 100644
--- a/website/versioned_docs/version-0.5.1/querying_data.md
+++ b/website/versioned_docs/version-0.5.1/querying_data.md
@@ -154,7 +154,7 @@ spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hu
```
For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.5.1/quick-start-guide.md b/website/versioned_docs/version-0.5.1/quick-start-guide.md
index 90a8bec85b8..a39dc627aba 100644
--- a/website/versioned_docs/version-0.5.1/quick-start-guide.md
+++ b/website/versioned_docs/version-0.5.1/quick-start-guide.md
@@ -75,7 +75,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
diff --git a/website/versioned_docs/version-0.5.1/writing_data.md b/website/versioned_docs/version-0.5.1/writing_data.md
index a6c40b65a11..3b023898ede 100644
--- a/website/versioned_docs/version-0.5.1/writing_data.md
+++ b/website/versioned_docs/version-0.5.1/writing_data.md
@@ -242,10 +242,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.5.2/deployment.md b/website/versioned_docs/version-0.5.2/deployment.md
index ad4843f0639..af83d1c3752 100644
--- a/website/versioned_docs/version-0.5.2/deployment.md
+++ b/website/versioned_docs/version-0.5.2/deployment.md
@@ -29,9 +29,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -130,7 +130,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.5.2/querying_data.md b/website/versioned_docs/version-0.5.2/querying_data.md
index 7887c888e1f..98b507b64bb 100644
--- a/website/versioned_docs/version-0.5.2/querying_data.md
+++ b/website/versioned_docs/version-0.5.2/querying_data.md
@@ -155,8 +155,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+For examples, refer to [Setup spark-shell in quickstart](quick-start-guide#setup).
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.5.2/quick-start-guide.md b/website/versioned_docs/version-0.5.2/quick-start-guide.md
index dee78296164..8a76b8189f7 100644
--- a/website/versioned_docs/version-0.5.2/quick-start-guide.md
+++ b/website/versioned_docs/version-0.5.2/quick-start-guide.md
@@ -75,7 +75,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
diff --git a/website/versioned_docs/version-0.5.2/writing_data.md b/website/versioned_docs/version-0.5.2/writing_data.md
index d5a7771ab72..82c86111c35 100644
--- a/website/versioned_docs/version-0.5.2/writing_data.md
+++ b/website/versioned_docs/version-0.5.2/writing_data.md
@@ -242,10 +242,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.5.3/deployment.md b/website/versioned_docs/version-0.5.3/deployment.md
index 018a21b5e21..aeeea7b24d3 100644
--- a/website/versioned_docs/version-0.5.3/deployment.md
+++ b/website/versioned_docs/version-0.5.3/deployment.md
@@ -29,9 +29,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -130,7 +130,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.5.3/querying_data.md b/website/versioned_docs/version-0.5.3/querying_data.md
index bec2c13f152..845478d4a6f 100644
--- a/website/versioned_docs/version-0.5.3/querying_data.md
+++ b/website/versioned_docs/version-0.5.3/querying_data.md
@@ -154,8 +154,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+For examples, refer to [Setup spark-shell in quickstart](quick-start-guide#setup).
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.5.3/quick-start-guide.md b/website/versioned_docs/version-0.5.3/quick-start-guide.md
index a2a10bd5146..b3eca64655b 100644
--- a/website/versioned_docs/version-0.5.3/quick-start-guide.md
+++ b/website/versioned_docs/version-0.5.3/quick-start-guide.md
@@ -79,7 +79,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
@@ -285,7 +285,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
diff --git a/website/versioned_docs/version-0.5.3/writing_data.md b/website/versioned_docs/version-0.5.3/writing_data.md
index 85aebf23c9d..cbec1ae4eff 100644
--- a/website/versioned_docs/version-0.5.3/writing_data.md
+++ b/website/versioned_docs/version-0.5.3/writing_data.md
@@ -242,10 +242,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.6.0/deployment.md b/website/versioned_docs/version-0.6.0/deployment.md
index 8112b63ac11..54a5c52be7d 100644
--- a/website/versioned_docs/version-0.6.0/deployment.md
+++ b/website/versioned_docs/version-0.6.0/deployment.md
@@ -29,9 +29,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -130,7 +130,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.6.0/querying_data.md b/website/versioned_docs/version-0.6.0/querying_data.md
index 412be1c38a1..766ada39fb8 100644
--- a/website/versioned_docs/version-0.6.0/querying_data.md
+++ b/website/versioned_docs/version-0.6.0/querying_data.md
@@ -166,8 +166,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+For examples, refer to [Setup spark-shell in quickstart](quick-start-guide#setup).
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.6.0/quick-start-guide.md b/website/versioned_docs/version-0.6.0/quick-start-guide.md
index 9bf766915cd..072da7c8661 100644
--- a/website/versioned_docs/version-0.6.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.6.0/quick-start-guide.md
@@ -79,7 +79,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
@@ -290,7 +290,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
diff --git a/website/versioned_docs/version-0.6.0/writing_data.md b/website/versioned_docs/version-0.6.0/writing_data.md
index 0a5b5933917..5253d6bb085 100644
--- a/website/versioned_docs/version-0.6.0/writing_data.md
+++ b/website/versioned_docs/version-0.6.0/writing_data.md
@@ -380,10 +380,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.7.0/deployment.md b/website/versioned_docs/version-0.7.0/deployment.md
index ba1c3a16fec..a24cdc248e8 100644
--- a/website/versioned_docs/version-0.7.0/deployment.md
+++ b/website/versioned_docs/version-0.7.0/deployment.md
@@ -29,9 +29,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -130,7 +130,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.7.0/querying_data.md b/website/versioned_docs/version-0.7.0/querying_data.md
index 8d227dc7bcf..5657ae107ab 100644
--- a/website/versioned_docs/version-0.7.0/querying_data.md
+++ b/website/versioned_docs/version-0.7.0/querying_data.md
@@ -166,8 +166,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+For examples, refer to [Setup spark-shell in quickstart](quick-start-guide#setup).
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.7.0/quick-start-guide.md b/website/versioned_docs/version-0.7.0/quick-start-guide.md
index 55719b0746e..fcc69aaacc5 100644
--- a/website/versioned_docs/version-0.7.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.7.0/quick-start-guide.md
@@ -79,7 +79,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
@@ -362,7 +362,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
\{: .notice--info}
## Query data
diff --git a/website/versioned_docs/version-0.7.0/writing_data.md b/website/versioned_docs/version-0.7.0/writing_data.md
index fb100aeb8f5..7750c93f7f4 100644
--- a/website/versioned_docs/version-0.7.0/writing_data.md
+++ b/website/versioned_docs/version-0.7.0/writing_data.md
@@ -382,10 +382,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.8.0/concurrency_control.md b/website/versioned_docs/version-0.8.0/concurrency_control.md
index aafaf5807ad..a3789940e91 100644
--- a/website/versioned_docs/version-0.8.0/concurrency_control.md
+++ b/website/versioned_docs/version-0.8.0/concurrency_control.md
@@ -20,13 +20,13 @@ between multiple table service writers and readers. Additionally, using MVCC, Hu
the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits (or writers) happening to the same table, if they do not have writes to overlapping files being changed, both writers are allowed to succeed.
This feature is currently *experimental* and requires either Zookeeper or HiveMetastore to acquire locks.
-It may be helpful to understand the different guarantees provided by [write operations](/docs/writing_data#write-operations) via Hudi datasource or the delta streamer.
+It may be helpful to understand the different guarantees provided by [write operations](writing_data#write-operations) via Hudi datasource or the delta streamer.
## Single Writer Guarantees
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
- - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
- - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](configurations#insert_drop_dups_opt_key) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](configurations#insert_drop_dups_opt_key)) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order.
## Multi Writer Guarantees
@@ -34,8 +34,8 @@ It may be helpful to understand the different guarantees provided by [write oper
With multiple writers using OCC, some of the above guarantees change as follows
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
-- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
-- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS_OPT_KEY) is enabled.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#insert_drop_dups_opt_key)) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#insert_drop_dups_opt_key)) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out of order due to multiple writer jobs finishing at different times.
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.8.0/deployment.md b/website/versioned_docs/version-0.8.0/deployment.md
index e25b5dab4c7..1eba743a957 100644
--- a/website/versioned_docs/version-0.8.0/deployment.md
+++ b/website/versioned_docs/version-0.8.0/deployment.md
@@ -29,9 +29,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -130,7 +130,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.8.0/querying_data.md b/website/versioned_docs/version-0.8.0/querying_data.md
index 75df56f2b52..450b96ba66b 100644
--- a/website/versioned_docs/version-0.8.0/querying_data.md
+++ b/website/versioned_docs/version-0.8.0/querying_data.md
@@ -168,8 +168,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+For examples, refer to [Setup spark-shell in quickstart](quick-start-guide#setup).
+Please refer to [configurations](/docs/configurations#spark-datasource-configs) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.8.0/quick-start-guide.md b/website/versioned_docs/version-0.8.0/quick-start-guide.md
index 3a7c7a24417..0b5d1e0d230 100644
--- a/website/versioned_docs/version-0.8.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.8.0/quick-start-guide.md
@@ -182,7 +182,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
:::
## Query data
diff --git a/website/versioned_docs/version-0.8.0/writing_data.md b/website/versioned_docs/version-0.8.0/writing_data.md
index fd2d5094f7a..75ad24fc26f 100644
--- a/website/versioned_docs/version-0.8.0/writing_data.md
+++ b/website/versioned_docs/version-0.8.0/writing_data.md
@@ -416,10 +416,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#logfilemaxsizelogfilesize--1gb) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.
diff --git a/website/versioned_docs/version-0.9.0/concurrency_control.md b/website/versioned_docs/version-0.9.0/concurrency_control.md
index 3a816b6d089..8186e64013a 100644
--- a/website/versioned_docs/version-0.9.0/concurrency_control.md
+++ b/website/versioned_docs/version-0.9.0/concurrency_control.md
@@ -19,13 +19,13 @@ between multiple table service writers and readers. Additionally, using MVCC, Hu
the same Hudi Table. Hudi supports `file level OCC`, i.e., for any 2 commits (or writers) happening to the same table, if they do not have writes to overlapping files being changed, both writers are allowed to succeed.
This feature is currently *experimental* and requires either Zookeeper or HiveMetastore to acquire locks.
-It may be helpful to understand the different guarantees provided by [write operations](/docs/writing_data#write-operations) via Hudi datasource or the delta streamer.
+It may be helpful to understand the different guarantees provided by [write operations](writing_data#write-operations) via Hudi datasource or the delta streamer.
## Single Writer Guarantees
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
- - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS) is enabled.
- - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](/docs/configurations#INSERT_DROP_DUPS) is enabled.
+ - *INSERT Guarantee*: The target table wilL NEVER have duplicates if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
+ - *BULK_INSERT Guarantee*: The target table will NEVER have duplicates if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints are NEVER out of order.
## Multi Writer Guarantees
@@ -33,8 +33,8 @@ It may be helpful to understand the different guarantees provided by [write oper
With multiple writers using OCC, some of the above guarantees change as follows
- *UPSERT Guarantee*: The target table will NEVER show duplicates.
-- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS) is enabled.
-- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](/docs/configurations#INSERT_DROP_DUPS) is enabled.
+- *INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
+- *BULK_INSERT Guarantee*: The target table MIGHT have duplicates even if [dedup](configurations#hoodiedatasourcewriteinsertdropduplicates) is enabled.
- *INCREMENTAL PULL Guarantee*: Data consumption and checkpoints MIGHT be out of order due to multiple writer jobs finishing at different times.
## Enabling Multi Writing
diff --git a/website/versioned_docs/version-0.9.0/deployment.md b/website/versioned_docs/version-0.9.0/deployment.md
index f5eba2c1ab1..5d954b4140a 100644
--- a/website/versioned_docs/version-0.9.0/deployment.md
+++ b/website/versioned_docs/version-0.9.0/deployment.md
@@ -27,9 +27,9 @@ With Merge_On_Read Table, Hudi ingestion needs to also take care of compacting d
### DeltaStreamer
-[DeltaStreamer](/docs/writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
+[DeltaStreamer](writing_data#deltastreamer) is the standalone utility to incrementally pull upstream changes from varied sources such as DFS, Kafka and DB Changelogs and ingest them to hudi tables. It runs as a spark application in 2 modes.
- - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
+ - **Run Once Mode** : In this mode, Deltastreamer performs one ingestion round which includes incrementally pulling events from upstream sources and ingesting them to hudi table. Background operations like cleaning old file versions and archiving hoodie timeline are automatically executed as part of the run. For Merge-On-Read tables, Compaction is also run inline as part of ingestion unless disabled by passing the flag "--disable-compaction". By default, Compaction is run inline for eve [...]
Here is an example invocation for reading from kafka topic in a single-run mode and writing to Merge On Read table type in a yarn cluster.
@@ -128,7 +128,7 @@ Here is an example invocation for reading from kafka topic in a continuous mode
### Spark Datasource Writer Jobs
-As described in [Writing Data](/docs/writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.ma [...]
+As described in [Writing Data](writing_data#datasource-writer), you can use spark datasource to ingest to hudi table. This mechanism allows you to ingest any spark dataframe in Hudi format. Hudi Spark DataSource also supports spark streaming to ingest a streaming source to Hudi table. For Merge On Read table types, inline compaction is turned on by default which runs after every ingestion run. The compaction frequency can be changed by setting the property "hoodie.compact.inline.max.delt [...]
Here is an example invocation using spark datasource
diff --git a/website/versioned_docs/version-0.9.0/querying_data.md b/website/versioned_docs/version-0.9.0/querying_data.md
index ede7d04f4d9..f5e37692178 100644
--- a/website/versioned_docs/version-0.9.0/querying_data.md
+++ b/website/versioned_docs/version-0.9.0/querying_data.md
@@ -173,8 +173,8 @@ hudiIncQueryDF.createOrReplaceTempView("hudi_trips_incremental")
spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_trips_incremental where fare > 20.0").show()
```
-For examples, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide#setup-spark-shell).
-Please refer to [configurations](/docs/configurations#spark-datasource) section, to view all datasource options.
+For examples, refer to [Setup spark-shell in quickstart](quick-start-guide#setup).
+Please refer to [configurations](/docs/configurations/#SPARK_DATASOURCE) section, to view all datasource options.
Additionally, `HoodieReadClient` offers the following functionality using Hudi's implicit indexing.
diff --git a/website/versioned_docs/version-0.9.0/quick-start-guide.md b/website/versioned_docs/version-0.9.0/quick-start-guide.md
index 3ab254501eb..a345030baff 100644
--- a/website/versioned_docs/version-0.9.0/quick-start-guide.md
+++ b/website/versioned_docs/version-0.9.0/quick-start-guide.md
@@ -368,7 +368,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
:::
</TabItem>
@@ -404,7 +404,7 @@ You can check the data generated under `/tmp/hudi_trips_cow/<region>/<country>/<
[Modeling data stored in Hudi](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113709185#FAQ-HowdoImodelthedatastoredinHudi)
and for info on ways to ingest data into Hudi, refer to [Writing Hudi Tables](/docs/writing_data).
Here we are using the default write operation : `upsert`. If you have a workload without updates, you can also issue
-`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](/docs/writing_data#write-operations)
+`insert` or `bulk_insert` operations which could be faster. To know more, refer to [Write operations](writing_data#write-operations)
:::
</TabItem>
diff --git a/website/versioned_docs/version-0.9.0/writing_data.md b/website/versioned_docs/version-0.9.0/writing_data.md
index 7671593bacf..8f95514ef73 100644
--- a/website/versioned_docs/version-0.9.0/writing_data.md
+++ b/website/versioned_docs/version-0.9.0/writing_data.md
@@ -415,10 +415,10 @@ column statistics etc. Even on some cloud data stores, there is often cost to li
Here are some ways to efficiently manage the storage of your Hudi tables.
- - The [small file handling feature](/docs/configurations#compactionSmallFileSize) in Hudi, profiles incoming workload
+ - The [small file handling feature](/docs/configurations#hoodieparquetsmallfilelimit) in Hudi, profiles incoming workload
and distributes inserts to existing file groups instead of creating new file groups, which can lead to small files.
- - Cleaner can be [configured](/docs/configurations#retainCommits) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
- - User can also tune the size of the [base/parquet file](/docs/configurations#limitFileSize), [log files](/docs/configurations#logFileMaxSize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
+ - Cleaner can be [configured](configurations#retaincommitsno_of_commits_to_retain--24) to clean up older file slices, more or less aggressively depending on maximum time for queries to run & lookback needed for incremental pull
+ - User can also tune the size of the [base/parquet file](/docs/configurations#hoodieparquetmaxfilesize), [log files](configurations#hoodielogfilemaxsize) & expected [compression ratio](/docs/configurations#parquetCompressionRatio),
such that sufficient number of inserts are grouped into the same file group, resulting in well sized base files ultimately.
- Intelligently tuning the [bulk insert parallelism](/docs/configurations#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups
once created cannot be deleted, but simply expanded as explained before.