[hudi] branch asf-site updated: [DOCS] More improvements to spec - Removing the draft status - Added table/details around table versioning/compatibility - 1 missing point on reader expectations. - Explain table configs and the hoodie.properties file!

[pr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

prasanna pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 3c93f9c636 [DOCS] More improvements to spec  - Removing the draft status  - Added table/details around table versioning/compatibility  - 1 missing point on reader expectations.  - Explain table configs and the hoodie.properties file!
     new e3d0d3ecf0 Merge pull request #6447 from vinothchandar/asf-site
3c93f9c636 is described below

commit 3c93f9c63618d193ea258b1486fe535e816ae514
Author: Vinoth Chandar <vi...@apache.org>
AuthorDate: Fri Aug 19 11:52:51 2022 +0530

    [DOCS] More improvements to spec
     - Removing the draft status
     - Added table/details around table versioning/compatibility
     - 1 missing point on reader expectations.
     - Explain table configs and the hoodie.properties file!
---
 website/docusaurus.config.js    |  5 +++++
 website/src/pages/tech-specs.md | 26 ++++++++++++++++++++------
 2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index 3ad2b8efad..d67bde016f 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -109,6 +109,7 @@ module.exports = {
     ],
   ],
   themeConfig: {
+    metadata: [{name: 'keywords', content: 'apache hudi, data lake, lakehouse, big data, apache spark, apache flink, presto, trino, analytics, data engineering'}],
     algolia: {
       apiKey: 'e300f1558b703c001c515c0e7f8e0908',
       indexName: 'apache_hudi',
@@ -136,6 +137,10 @@ module.exports = {
               label: 'FAQ',
               href: '/docs/faq',
             },
+            {
+              label: 'Tech Specs',
+              href: '/tech-specs',
+            },
             {
               label: 'Technical Wiki',
               href: 'https://cwiki.apache.org/confluence/display/HUDI',
diff --git a/website/src/pages/tech-specs.md b/website/src/pages/tech-specs.md
index a5d8b9fa46..a065ac99b4 100644
--- a/website/src/pages/tech-specs.md
+++ b/website/src/pages/tech-specs.md
@@ -1,4 +1,10 @@
-# Apache Hudi Technical Specification [DRAFT]
+# Apache Hudi Technical Specification
+
+
+| Syntax        | Description |
+|:--------------|:-----------:|
+| Last Updated  |  Aug 2022   |
+| [Table Version](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableVersion.java#L29) |      4      |
 
 
 This document is a specification for the Hudi's Storage Format which transforms immutable cloud/file storage systems into transactional data lakes. 
@@ -25,13 +31,16 @@ This document is intended as reference guide for any compute engines, that aim t
 
 At a high level, Hudi organizes data into a directory structure under the base path (root directory for the Hudi table). The directory structure can be flat (non-partitioned) or based on coarse-grained partitioning values set for the table. Non-partitioned tables store all the data files under the base path. 
 Note that, unlike Hive style partitioning, partition columns are not removed from data files and partitioning is a mere organization of data files. A special reserved *.hoodie* directory under the base path is used to store transaction logs and metadata.
+A special file `hoodie.properties` persists table level configurations, shared by writers and readers of the table. These configurations are explained [here](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java), 
+and any config without a default value needs to be specified during table creation.
 
-	/data/hudi_trips/ 					<== BASE PATH
-	├── .hoodie/      					<== META PATH
-	│   └── metadata/ 
+	/data/hudi_trips/ 					<== Base Path
+	├── .hoodie/      					<== Meta Path
+    |   └── hoodie.properties           <== Table Configs
+	│   └── metadata/                   <== Table Metadata
 	├── americas/
 	│   ├── brazil/
-	│   │   └── sao_paulo/				<== PARTITION PATH 
+	│   │   └── sao_paulo/				<== Partition Path 
 	│   │       ├── <data_files>
 	│   └── united_states/
 	│       └── san_francisco/
@@ -269,7 +278,7 @@ Readers will use snapshot isolation to query a Hudi table at a consistent point
    1. Contents of the log files should be loaded into an effective point lookup data structure (in-memory or persisted)
    2. Duplicate record keys should be merged based on the ordering field specified. It is important to order the inserts and deletes in the right order to be consistent and idempotent. 
    3. When the base file is scanned, for every record block, the reader has to lookup if there is a newer version of the data available for the record keys in the block and merge them into the record iterator. 
-
+4. In addition, all the "replacecommit" metadata needs to read to filter out flle groups, that have been replaced with newer file groups, by actions like clustering.
 
 
 ## Writer Expectations
@@ -286,6 +295,11 @@ Writer into Hudi will have to ingest new records, updates to existing records or
 6. Synchronizing Indexes and metadata needs to be done in the same transaction that commits the modifications to the table. 
 
 
+## Compatibility
+
+Compatibility between different readers and writers is enforced through the `hoodie.table.version` table config. Hudi storage format evolves in a backwards compatible way for readers, where newer readers
+can read older table versions correctly. However older readers may be required to upgrade in order to read higher table versions. Hence, we recommend upgrading readers first, before upgrading writers when
+the table version evolves.
 
 ## Balancing write and query performance