You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vi...@apache.org on 2019/03/13 22:41:16 UTC
[incubator-hudi-site] 13/19: Pushing refreshed site content
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hudi-site.git
commit dec68b2816fd79eb2e58b09c8c1606e8e61e95f6
Author: Vinoth Chandar <vi...@uber.com>
AuthorDate: Mon Feb 25 12:58:29 2019 -0800
Pushing refreshed site content
- Reflects footer, privacy disclaimer changes
- Reflects various fixes to image links
- Reflects new contribution, community links
---
content/404.html | 31 +-
content/admin_guide.html | 83 +++--
content/api_docs.html | 399 --------------------
content/community.html | 98 +++--
content/comparison.html | 31 +-
content/concepts.html | 53 +--
content/configurations.html | 91 +++--
content/{dev_setup.html => contributing.html} | 154 ++++++--
content/css/customstyles.css | 4 +-
content/css/lavish-bootstrap.css | 8 +
content/css/theme-blue.css | 3 +-
content/feed.xml | 20 +-
content/gcs_hoodie.html | 31 +-
content/images/asf_logo.svg | 210 +++++++++++
content/images/hoodie_cow.png | Bin 31136 -> 0 bytes
content/images/hoodie_intro_1.png | Bin 23478 -> 27890 bytes
content/images/hoodie_mor.png | Bin 56002 -> 0 bytes
content/images/hudi_cow.png | Bin 0 -> 48994 bytes
content/images/hudi_mor.png | Bin 0 -> 92073 bytes
.../{hoodie_timeline.png => hudi_timeline.png} | Bin
content/implementation.html | 142 ++++----
content/incremental_processing.html | 38 +-
content/index.html | 55 +--
content/js/mydoc_scroll.html | 31 +-
content/migration_guide.html | 154 ++++----
content/news.html | 45 ++-
content/news_archive.html | 42 ++-
content/powered_by.html | 31 +-
content/{code_and_design.html => privacy.html} | 114 ++----
content/quickstart.html | 186 ++++++----
content/roadmap.html | 405 ---------------------
content/s3_hoodie.html | 31 +-
content/search.json | 60 ++-
content/sitemap.xml | 20 +-
content/sql_queries.html | 37 +-
content/strata-talk.html | 44 ++-
content/use_cases.html | 31 +-
docs/_includes/footer.html | 2 +-
38 files changed, 1218 insertions(+), 1466 deletions(-)
diff --git a/content/404.html b/content/404.html
index 5c95e2c..9491810 100644
--- a/content/404.html
+++ b/content/404.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -379,9 +380,19 @@ $('#toc').on('click', 'a', function() {
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/admin_guide.html b/content/admin_guide.html
index 72e5af4..470a219 100644
--- a/content/admin_guide.html
+++ b/content/admin_guide.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -379,8 +380,9 @@ hoodie->create --path /user/hive/warehouse/table1 --tableName hoodie_table_1
</code></pre>
</div>
-<p>To see the description of hoodie table, use the command:
-<code class="highlighter-rouge">
+<p>To see the description of hoodie table, use the command:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>
hoodie:hoodie_table_1->desc
18/09/06 15:57:19 INFO timeline.HoodieActiveTimeline: Loaded instants []
_________________________________________________________
@@ -392,7 +394,9 @@ hoodie:hoodie_table_1->desc
| hoodie.table.name | hoodie_table_1 |
| hoodie.table.type | COPY_ON_WRITE |
| hoodie.archivelog.folder| |
-</code></p>
+
+</code></pre>
+</div>
<p>Following is a sample command to connect to a Hoodie dataset contains uber trips.</p>
@@ -511,7 +515,7 @@ order (See Concepts). The below commands allow users to view the file-slices for
| Partition | FileId | Base-Instant | Data-File | Data-File Size| Num Delta Files| Total Delta Size| Delta Size - compaction scheduled| Delta Size - compaction unscheduled| Delta To Base Ratio - compaction scheduled| Delta To Base Ratio - compaction unscheduled| Delta Files - compaction scheduled | Delta Files - compaction unscheduled|
|========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== [...]
| 2018/08/31| 111415c3-f26d-4639-86c8-f9956f245ac3| 20181002180759| hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/111415c3-f26d-4639-86c8-f9956f245ac3_0_20181002180759.parquet| 432.5 KB | 1 | 20.8 KB | 20.8 KB | 0.0 B | 0.0 B | 0.0 B | [HoodieLogFile {hdfs://namenode:8020/user/hive/warehouse/stock_ticks_mor/2018/08/31/.111415c3-f26d-4639-86c8-f9956f245ac3_20181002180759.log.1}]| [] |
-
+
hoodie:stock_ticks_mor->
</code></pre>
</div>
@@ -548,7 +552,7 @@ This is a sequence file that contains a mapping from commitNumber => json wit
<h4 id="compactions">Compactions</h4>
-<p>To get an idea of the lag between compaction and writer applications, use the below command to list down all
+<p>To get an idea of the lag between compaction and writer applications, use the below command to list down all
pending compactions.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>hoodie:trips->compactions show all
@@ -639,7 +643,7 @@ Description: Run Compaction for given instant time
...
COMPACTION PLAN VALID
-
+
___________________________________________________________________________________________________________________________________________________________________________________________________________________________
| File Id | Base Instant Time| Base Data File | Num Delta Files| Valid| Error|
|==========================================================================================================================================================================================================================|
@@ -664,13 +668,14 @@ hoodie:stock_ticks_mor->compaction validate --instant 20181005222601
<p>The following commands must be executed without any other writer/ingestion application running.</p>
-<p>Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
-operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
+<p>Sometimes, it becomes necessary to remove a fileId from a compaction-plan inorder to speed-up or unblock compaction
+operation. Any new log-files that happened on this file after the compaction got scheduled will be safely renamed
so that are preserved. Hudi provides the following CLI to support it</p>
<h5 id="unscheduling-compaction">UnScheduling Compaction</h5>
-<div class="highlighter-rouge"><pre class="highlight"><code>hoodie:trips->compaction unscheduleFileId --fileId <FileUUID>
+<div class="highlighter-rouge"><pre class="highlight"><code>
+hoodie:trips->compaction unscheduleFileId --fileId <FileUUID>
....
No File renames needed to unschedule file from pending compaction. Operation successful.
@@ -678,27 +683,31 @@ No File renames needed to unschedule file from pending compaction. Operation suc
</div>
<p>In other cases, an entire compaction plan needs to be reverted. This is supported by the following CLI
-<code class="highlighter-rouge">
-hoodie:trips->compaction unschedule --compactionInstant <compactionInstant>
+```</p>
+
+<p>hoodie:trips->compaction unschedule –compactionInstant <compactionInstant>
.....
-No File renames needed to unschedule pending compaction. Operation successful.
-</code></p>
+No File renames needed to unschedule pending compaction. Operation successful.</compactionInstant></p>
-<h5 id="repair-compaction">Repair Compaction</h5>
+<div class="highlighter-rouge"><pre class="highlight"><code>
+##### Repair Compaction
-<p>The above compaction unscheduling operations could sometimes fail partially (e:g -> HDFS temporarily unavailable). With
-partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
-<code class="highlighter-rouge">compaction validate</code>, you can notice invalid compaction operations if there is one. In these cases, the repair
+The above compaction unscheduling operations could sometimes fail partially (e:g -> HDFS temporarily unavailable). With
+partial failures, the compaction operation could become inconsistent with the state of file-slices. When you run
+`compaction validate`, you can notice invalid compaction operations if there is one. In these cases, the repair
command comes to the rescue, it will rearrange the file-slices so that there is no loss and the file-slices are
-consistent with the compaction plan</p>
+consistent with the compaction plan
-<div class="highlighter-rouge"><pre class="highlight"><code>hoodie:stock_ticks_mor->compaction repair --instant 20181005222611
-......
-Compaction successfully repaired
-.....
</code></pre>
</div>
+<p>hoodie:stock_ticks_mor->compaction repair –instant 20181005222611
+……
+Compaction successfully repaired
+…..</p>
+
+<p>```</p>
+
<h2 id="metrics">Metrics</h2>
<p>Once the Hoodie Client is configured with the right datasetname and environment for metrics, it produces the following graphite metrics, that aid in debugging hoodie datasets</p>
@@ -785,9 +794,19 @@ In the future, a more sophisticated debug/management UI would be added to the pr
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/api_docs.html b/content/api_docs.html
deleted file mode 100644
index 2898b67..0000000
--- a/content/api_docs.html
+++ /dev/null
@@ -1,399 +0,0 @@
-<!DOCTYPE html>
-<head>
- <meta charset="utf-8">
-<meta http-equiv="X-UA-Compatible" content="IE=edge">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<meta name="description" content="">
-<meta name="keywords" content=" usecases">
-<title>API Docs | Hudi</title>
-<link rel="stylesheet" href="css/syntax.css">
-
-
-<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
-<!--<link rel="stylesheet" type="text/css" href="css/bootstrap.min.css">-->
-<link rel="stylesheet" href="css/modern-business.css">
-<link rel="stylesheet" href="css/lavish-bootstrap.css">
-<link rel="stylesheet" href="css/customstyles.css">
-<link rel="stylesheet" href="css/theme-blue.css">
-
-<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
-<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js"></script>
-<script src="js/jquery.navgoco.min.js"></script>
-
-
-<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
-<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js"></script>
-<script src="js/toc.js"></script>
-<script src="js/customscripts.js"></script>
-
-<script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
-
- ga('create', 'UA-93561550-1', 'auto');
- ga('send', 'pageview');
-
-</script>
-
-<link rel="shortcut icon" href="images/favicon.ico">
-
-<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
-<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
-<!--[if lt IE 9]>
-<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
-<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
-<![endif]-->
-
-<link rel="alternate" type="application/rss+xml" title="" href="http://0.0.0.0:4000feed.xml">
-
- <script>
- $(document).ready(function() {
- // Initialize navgoco with default options
- $("#mysidebar").navgoco({
- caretHtml: '',
- accordion: true,
- openClass: 'active', // open
- save: false, // leave false or nav highlighting doesn't work right
- cookie: {
- name: 'navgoco',
- expires: false,
- path: '/'
- },
- slide: {
- duration: 400,
- easing: 'swing'
- }
- });
-
- $("#collapseAll").click(function(e) {
- e.preventDefault();
- $("#mysidebar").navgoco('toggle', false);
- });
-
- $("#expandAll").click(function(e) {
- e.preventDefault();
- $("#mysidebar").navgoco('toggle', true);
- });
-
- });
-
- </script>
- <script>
- $(function () {
- $('[data-toggle="tooltip"]').tooltip()
- })
- </script>
-
-
-</head>
-<body>
-<!-- Navigation -->
-
-<nav class="navbar navbar-inverse navbar-fixed-top">
- <div class="container topnavlinks">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
- <img src="images/hudi_site_logo.png" alt="Hudi logo"/>
- <!--Hudi-->
- </span></a>
- </div>
- <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
- <ul class="nav navbar-nav navbar-right">
- <!-- entries without drop-downs appear here -->
-
-
-
- <li><a href="news">News</a></li>
-
-
-
- <li><a href="community.html">Community</a></li>
-
-
-
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
-
-
-
- <!-- entries with drop-downs appear here -->
- <!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="dev_setup.html">Setup</a></li>
-
-
-
- <li class="dropdownActive"><a href="api_docs.html">API Docs</a></li>
-
-
-
- <li><a href="code_and_design.html">Code Structure</a></li>
-
-
-
- <li><a href="roadmap.html">Roadmap</a></li>
-
-
- </ul>
- </li>
-
-
-
- <li>
-
-
-
- <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the API Docs page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
-
-<li>
-
-
- <!--comment out this block if you want to hide search-->
- <li>
- <!--start search-->
- <div id="search-demo-container">
- <input type="text" id="search-input" placeholder="search...">
- <ul id="results-container"></ul>
- </div>
- <script src="js/jekyll-search.js" type="text/javascript"></script>
- <script type="text/javascript">
- SimpleJekyllSearch.init({
- searchInput: document.getElementById('search-input'),
- resultsContainer: document.getElementById('results-container'),
- dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}" title="API Docs">{title}</a></li>',
- noResultsText: 'No results found.',
- limit: 10,
- fuzzy: true,
- })
- </script>
- <!--end search-->
- </li>
- </ul>
- </div>
- </div>
- <!-- /.container -->
-</nav>
-
-<!-- Page Content -->
-<div class="container">
- <div class="col-lg-12"> </div>
- <!-- Content Row -->
- <div class="row">
- <!-- Sidebar Column -->
- <div class="col-md-3">
-
-
-
-
-
-
-
-
-
-
-<ul id="mysidebar" class="nav">
- <li class="sidebarTitle">Latest Version</li>
-
-
-
- <li>
- <a href="#">Getting Started</a>
- <ul>
-
-
-
- <li><a href="index.html">Overview</a></li>
-
-
-
-
-
-
- <li><a href="quickstart.html">Quickstart</a></li>
-
-
-
-
-
-
- <li><a href="use_cases.html">Use Cases</a></li>
-
-
-
-
-
-
- <li><a href="powered_by.html">Talks & Powered By</a></li>
-
-
-
-
- </ul>
-
-
-
- <li>
- <a href="#">Documentation</a>
- <ul>
-
-
-
- <li><a href="concepts.html">Concepts</a></li>
-
-
-
-
-
-
- <li><a href="implementation.html">Implementation</a></li>
-
-
-
-
-
-
- <li><a href="configurations.html">Configurations</a></li>
-
-
-
-
-
-
- <li><a href="sql_queries.html">SQL Queries</a></li>
-
-
-
-
-
-
- <li><a href="migration_guide.html">Migration Guide</a></li>
-
-
-
-
-
-
- <li><a href="incremental_processing.html">Incremental Processing</a></li>
-
-
-
-
-
-
- <li><a href="admin_guide.html">Admin Guide</a></li>
-
-
-
-
-
-
- <li><a href="comparison.html">Comparison</a></li>
-
-
-
-
- </ul>
-
-
-
- <!-- if you aren't using the accordion, uncomment this block:
- <p class="external">
- <a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
- </p>
- -->
- </li>
-</ul>
-</div>
-
-<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
-<script>$("li.active").parents('li').toggleClass("active");</script>
-
- <!-- Content Column -->
- <div class="col-md-9">
- <div class="post-header">
- <h1 class="post-title-main">API Docs</h1>
-</div>
-
-
-
-<div class="post-content">
-
-
-
-
-
-<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
-<script>
-$( document ).ready(function() {
- // Handler for .ready() called.
-
-$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
-
-/* this offset helps account for the space taken up by the floating toolbar. */
-$('#toc').on('click', 'a', function() {
- var target = $(this.getAttribute('href'))
- , scroll_target = target.offset().top
-
- $(window).scrollTop(scroll_target - 10);
- return false
-})
-
-});
-</script>
-
-<div id="toc"></div>
-
-
-
-
-
-
- <p>Work In Progress</p>
-
-
-
- <div class="tags">
-
- </div>
-
-
-
-</div>
-
-<hr class="shaded"/>
-
-<footer>
- <div class="row">
- <div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
- </div>
- </div>
-</footer>
-
-
- </div>
- <!-- /.row -->
-</div>
-<!-- /.container -->
- </div>
-
-</body>
-
-</html>
\ No newline at end of file
diff --git a/content/community.html b/content/community.html
index de173be..39488eb 100644
--- a/content/community.html
+++ b/content/community.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -340,20 +341,63 @@
- <h2 id="contributing">Contributing</h2>
-<p>We :heart: contributions. If you find a bug in the library or would like to add new features, go ahead and open
-issues or pull requests against this repo. Before you do so, please sign the
-<a href="https://www.apache.org/licenses/icla.pdf">Apache CLA</a>.
-Also, be sure to write unit tests for your bug fix or feature to show that it works as expected.
-If the reviewer feels this contributions needs to be in the release notes, please add it to CHANGELOG.md as well.</p>
-
-<p>If you want to participate in day-day conversations, please join our <a href="https://join.slack.com/t/apache-hudi/signup">slack group</a>.
-If you are from select pre-listed email domains, you can self signup. Others, please subscribe to dev@hudi.apache.org</p>
+ <h2 id="engage-with-us">Engage with us</h2>
+
+<p>There are several ways to get in touch with the Hudi community.</p>
+
+<table>
+ <thead>
+ <tr>
+ <th>When?</th>
+ <th>Channel to use</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>For any general questions, user support, development discussions</td>
+ <td>Dev Mailing list (<a href="mailto:dev-subscribe@hudi.apache.org">Subscribe</a>, <a href="mailto:dev-unsubscribe@hudi.apache.or& [...]
+ </tr>
+ <tr>
+ <td>For reporting bugs or issues or discover known issues</td>
+ <td>Please use <a href="https://issues.apache.org/jira/projects/HUDI/summary">ASF Hudi JIRA</a></td>
+ </tr>
+ <tr>
+ <td>For quick pings & 1-1 chats</td>
+ <td>Join our <a href="https://join.slack.com/t/apache-hudi/signup">slack group</a></td>
+ </tr>
+ <tr>
+ <td>For proposing large features, changes</td>
+ <td>Start a Hudi Improvement Process (HIP). Instructions coming soon.</td>
+ </tr>
+ <tr>
+ <td>For stream of commits, pull requests etc</td>
+ <td>Commits Mailing list (<a href="mailto:commits-subscribe@hudi.apache.org">Subscribe</a>, <a href="mailto:commits-unsubscribe@hudi.� [...]
+ </tr>
+ </tbody>
+</table>
+
+<p>If you wish to report a security vulnerability, please contact <a href="mailto:security@apache.org">security@apache.org</a>.
+Apache Hudi follows the typical Apache vulnerability handling <a href="https://apache.org/security/committers.html#vulnerability-handling">process</a>.</p>
+
+<h2 id="contributing">Contributing</h2>
+
+<p>Apache Hudi community welcomes contributions from anyone!</p>
+
+<p>Here are few ways, you can get involved.</p>
+
+<ul>
+ <li>Ask (and/or) answer questions on our support channels listed above.</li>
+ <li>Review code or HIPs</li>
+ <li>Help improve documentation</li>
+ <li>Testing; Improving out-of-box experience by reporting bugs</li>
+ <li>Share new ideas/directions to pursue or propose a new HIP</li>
+ <li>Contributing code to the project</li>
+</ul>
-<h2 id="becoming-a-committer">Becoming a Committer</h2>
+<h4 id="code-contributions">Code Contributions</h4>
-<p>Hoodie has adopted a lot of guidelines set forth in <a href="https://www.chromium.org/getting-involved/become-a-committer">Google Chromium project</a>, to determine committership proposals. However, given this is a much younger project, we would have the contribution bar to be 10-15 non-trivial patches instead.
-Additionally, we expect active engagement with the community over a few months, in terms of conference/meetup talks, helping out with issues/questions on slack/github.</p>
+<p>Useful resources for contributing can be found under the “Developers” top menu.
+Specifically, please refer to the detailed <a href="contributing.html">contribution guide</a>.</p>
<div class="tags">
@@ -369,9 +413,19 @@ Additionally, we expect active engagement with the community over a few months,
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/comparison.html b/content/comparison.html
index d56ab17..34082e0 100644
--- a/content/comparison.html
+++ b/content/comparison.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -404,9 +405,19 @@ and will eventually happen as a <a href="https://github.com/uber/hoodie/issues/8
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/concepts.html b/content/concepts.html
index cc498fb..7e85d32 100644
--- a/content/concepts.html
+++ b/content/concepts.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -357,7 +358,7 @@ Such key activities include</p>
<li><code class="highlighter-rouge">COMMITS</code> - A single commit captures information about an <strong>atomic write</strong> of a batch of records into a dataset.
Commits are identified by a monotonically increasing timestamp, denoting the start of the write operation.</li>
<li><code class="highlighter-rouge">CLEANS</code> - Background activity that gets rid of older versions of files in the dataset, that are no longer needed.</li>
- <li><code class="highlighter-rouge">DELTA_COMMITS</code> - A single commit captures information about an <strong>atomic write</strong> of a batch of records into a
+ <li><code class="highlighter-rouge">DELTA_COMMITS</code> - A single commit captures information about an <strong>atomic write</strong> of a batch of records into a
MergeOnRead storage type of dataset</li>
<li><code class="highlighter-rouge">COMPACTIONS</code> - Background activity to reconcile differential data structures within Hudi e.g: moving updates from row based log files to columnar formats.</li>
</ul>
@@ -375,15 +376,15 @@ only the changed files without say scanning all the time buckets > 07:00.</p>
<h2 id="terminologies">Terminologies</h2>
<ul>
- <li><code class="highlighter-rouge">Hudi Dataset</code>
+ <li><code class="highlighter-rouge">Hudi Dataset</code>
A structured hive/spark dataset managed by Hudi. Hudi supports both partitioned and non-partitioned Hive tables.</li>
- <li><code class="highlighter-rouge">Commit</code>
- A commit marks a new batch of data applied to a dataset. Hudi maintains monotonically increasing timestamps to track commits and guarantees that a commit is atomically
+ <li><code class="highlighter-rouge">Commit</code>
+ A commit marks a new batch of data applied to a dataset. Hudi maintains monotonically increasing timestamps to track commits and guarantees that a commit is atomically
published.</li>
<li><code class="highlighter-rouge">Commit Timeline</code>
Commit Timeline refers to the sequence of Commits that was applied in order on a dataset over its lifetime.</li>
- <li><code class="highlighter-rouge">File Slice</code>
- Hudi provides efficient handling of updates by having a fixed mapping between record key to a logical file Id.
+ <li><code class="highlighter-rouge">File Slice</code>
+ Hudi provides efficient handling of updates by having a fixed mapping between record key to a logical file Id.
Hudi uses MVCC to provide atomicity and isolation of readers from a writer. This means that a logical fileId will
have many physical versions of it. Each of these physical version of a file represents a complete view of the
file as of a commit and is called File Slice</li>
@@ -423,8 +424,6 @@ to the queries (i.e how data is read).</p>
<li>Merge On Read : Also provides a near-real time datasets in the order of 5 mins, by shifting some of the write cost, to the reads and merging incoming and on-disk data on-the-fly</li>
</ul>
-<div class="bs-callout bs-callout-info">Hudi is a young project. merge-on-read is currently underway. Get involved <a href="https://github.com/uber/Hudi/projects/1">here</a></div>
-
<p>Regardless of the storage type, Hudi organizes a datasets into a directory structure under a <code class="highlighter-rouge">basepath</code>,
very similar to Hive tables. Dataset is broken up into partitions, which are folders containing files for that partition.
Each partition uniquely identified by its <code class="highlighter-rouge">partitionpath</code>, which is relative to the basepath.</p>
@@ -444,11 +443,11 @@ commit, such that only columnar data exists. As a result, the write amplificatio
<p>Following illustrates how this works conceptually, when data written into copy-on-write storage and two queries running on top of it.</p>
-<figure><img class="docimage" src="images/Hudi_cow.png" alt="Hudi_cow.png" /></figure>
+<figure><img class="docimage" src="images/hudi_cow.png" alt="hudi_cow.png" /></figure>
<p>As data gets written, updates to existing file ids, produce a new version for that file id stamped with the commit and
inserts allocate a new file id and write its first version for that file id. These file versions and their commits are color coded above.
-Normal SQL queries running against such dataset (eg: select count(*) counting the total records in that partition), first checks the timeline for latest commit
+Normal SQL queries running against such dataset (eg: <code class="highlighter-rouge">select count(*)</code> counting the total records in that partition), first checks the timeline for latest commit
and filters all but latest versions of each file id. As you can see, an old query does not see the current inflight commit’s files colored in pink,
but a new query starting after the commit picks up the new data. Thus queries are immune to any write failures/partial writes and only run on committed data.</p>
@@ -470,7 +469,7 @@ their columnar base data, to keep the query performance in check (larger append
<p>Following illustrates how the storage works, and shows queries on both near-real time table and read optimized table.</p>
-<figure><img class="docimage" src="images/Hudi_mor.png" alt="Hudi_mor.png" style="max-width: 1000px" /></figure>
+<figure><img class="docimage" src="images/hudi_mor.png" alt="hudi_mor.png" style="max-width: 1000px" /></figure>
<p>There are lot of interesting things happening in this example, which bring out the subleties in the approach.</p>
@@ -488,8 +487,6 @@ their columnar base data, to keep the query performance in check (larger append
published within X minutes in a consistent fashion.</li>
</ul>
-<div class="bs-callout bs-callout-info">Hudi is a young project. merge-on-read is currently underway. Get involved <a href="https://github.com/uber/hoodie/projects/1">here</a></div>
-
<p>The intention of merge on read storage, is to enable near real-time processing directly on top of Hadoop, as opposed to copying
data out to specialized systems, which may not be able to handle the data volume.</p>
@@ -567,9 +564,19 @@ data out to specialized systems, which may not be able to handle the data volume
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/configurations.html b/content/configurations.html
index be91882..5f1adb8 100644
--- a/content/configurations.html
+++ b/content/configurations.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -488,7 +489,7 @@
Actual value ontained by invoking .toString()</span></li>
<li><a href="#KEYGENERATOR_CLASS_OPT_KEY">KEYGENERATOR_CLASS_OPT_KEY</a> (Default: com.uber.hoodie.SimpleKeyGenerator) <br />
<span style="color:grey">Key generator class, that implements will extract the key out of incoming <code class="highlighter-rouge">Row</code> object</span></li>
- <li><a href="#COMMIT_METADATA_KEYPREFIX_OPT_KEY">COMMIT_METADATA_KEYPREFIX_OPT_KEY</a> (Default: _) <br />
+ <li><a href="#COMMIT_METADATA_KEYPREFIX_OPT_KEY">COMMIT_METADATA_KEYPREFIX_OPT_KEY</a> (Default: <code class="highlighter-rouge">_</code>) <br />
<span style="color:grey">Option keys beginning with this prefix, are automatically added to the commit/deltacommit metadata.
This is useful to store checkpointing information, in a consistent way with the hoodie timeline</span></li>
</ul>
@@ -518,31 +519,33 @@
<p>Writing data via Hudi happens as a Spark job and thus general rules of spark debugging applies here too. Below is a list of things to keep in mind, if you are looking to improving performance or reliability.</p>
-<ul>
- <li><strong>Write operations</strong> : Use <code class="highlighter-rouge">bulkinsert</code> to load new data into a table, and there on use <code class="highlighter-rouge">upsert</code>/<code class="highlighter-rouge">insert</code>.
- Difference between them is that bulk insert uses a disk based write path to scale to load large inputs without need to cache it.</li>
- <li><strong>Input Parallelism</strong> : By default, Hoodie tends to over-partition input (i.e <code class="highlighter-rouge">withParallelism(1500)</code>), to ensure each Spark partition stays within the 2GB limit for inputs upto 500GB. Bump this up accordingly if you have larger inputs. We recommend having shuffle parallelism <code class="highlighter-rouge">hoodie.[insert|upsert|bulkinsert].shuffle.parallelism</code> such that its atleast input_data_size/500MB</li>
- <li><strong>Off-heap memory</strong> : Hoodie writes parquet files and that needs good amount of off-heap memory proportional to schema width. Consider setting something like <code class="highlighter-rouge">spark.yarn.executor.memoryOverhead</code> or <code class="highlighter-rouge">spark.yarn.driver.memoryOverhead</code>, if you are running into such failures.</li>
- <li><strong>Spark Memory</strong> : Typically, hoodie needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some <code class="highlighter-rouge">spark.storage.memoryFraction</code> will generally help boost performance.</li>
- <li><strong>Sizing files</strong> : Set <code class="highlighter-rouge">limitFileSize</code> above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.</li>
- <li><strong>Timeseries/Log data</strong> : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases
- <ul>
- <li>Consider tuning the bloom filter accuracy via <code class="highlighter-rouge">.bloomFilterFPP()/bloomFilterNumEntries()</code> to achieve your target index look up time</li>
- <li>Consider making a key that is prefixed with time of the event, which will enable range pruning & significantly speeding up index lookup.</li>
- </ul>
- </li>
- <li><strong>GC Tuning</strong> : Please be sure to follow garbage collection tuning tips from Spark tuning guide to avoid OutOfMemory errors
- <ul>
- <li>[Must] Use G1/CMS Collector. Sample CMS Flags to add to spark.executor.extraJavaOptions : ``-XX:NewSize=1g -XX:SurvivorRatio=2 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tm [...]
- <li>If it keeps OOMing still, reduce spark memory conservatively: <code class="highlighter-rouge">spark.memory.fraction=0.2, spark.memory.storageFraction=0.2</code> allowing it to spill rather than OOM. (reliably slow vs crashing intermittently)</li>
- </ul>
- </li>
-</ul>
+<p><strong>Write operations</strong> : Use <code class="highlighter-rouge">bulkinsert</code> to load new data into a table, and there on use <code class="highlighter-rouge">upsert</code>/<code class="highlighter-rouge">insert</code>.
+ Difference between them is that bulk insert uses a disk based write path to scale to load large inputs without need to cache it.</p>
+
+<p><strong>Input Parallelism</strong> : By default, Hoodie tends to over-partition input (i.e <code class="highlighter-rouge">withParallelism(1500)</code>), to ensure each Spark partition stays within the 2GB limit for inputs upto 500GB. Bump this up accordingly if you have larger inputs. We recommend having shuffle parallelism <code class="highlighter-rouge">hoodie.[insert|upsert|bulkinsert].shuffle.parallelism</code> such that its atleast input_data_size/500MB</p>
+
+<p><strong>Off-heap memory</strong> : Hoodie writes parquet files and that needs good amount of off-heap memory proportional to schema width. Consider setting something like <code class="highlighter-rouge">spark.yarn.executor.memoryOverhead</code> or <code class="highlighter-rouge">spark.yarn.driver.memoryOverhead</code>, if you are running into such failures.</p>
+
+<p><strong>Spark Memory</strong> : Typically, hoodie needs to be able to read a single file into memory to perform merges or compactions and thus the executor memory should be sufficient to accomodate this. In addition, Hoodie caches the input to be able to intelligently place data and thus leaving some <code class="highlighter-rouge">spark.storage.memoryFraction</code> will generally help boost performance.</p>
+
+<p><strong>Sizing files</strong> : Set <code class="highlighter-rouge">limitFileSize</code> above judiciously, to balance ingest/write latency vs number of files & consequently metadata overhead associated with it.</p>
+
+<p><strong>Timeseries/Log data</strong> : Default configs are tuned for database/nosql changelogs where individual record sizes are large. Another very popular class of data is timeseries/event/log data that tends to be more volumnious with lot more records per partition. In such cases
+ - Consider tuning the bloom filter accuracy via <code class="highlighter-rouge">.bloomFilterFPP()/bloomFilterNumEntries()</code> to achieve your target index look up time
+ - Consider making a key that is prefixed with time of the event, which will enable range pruning & significantly speeding up index lookup.</p>
+
+<p><strong>GC Tuning</strong> : Please be sure to follow garbage collection tuning tips from Spark tuning guide to avoid OutOfMemory errors
+[Must] Use G1/CMS Collector. Sample CMS Flags to add to spark.executor.extraJavaOptions :</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>-XX:NewSize=1g -XX:SurvivorRatio=2 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hoodie-heapdump.hprof
+</code></pre>
+</div>
+
+<p>If it keeps OOMing still, reduce spark memory conservatively: <code class="highlighter-rouge">spark.memory.fraction=0.2, spark.memory.storageFraction=0.2</code> allowing it to spill rather than OOM. (reliably slow vs crashing intermittently)</p>
<p>Below is a full working production config</p>
-<p>```
- spark.driver.extraClassPath /etc/hive/conf
+<div class="highlighter-rouge"><pre class="highlight"><code> spark.driver.extraClassPath /etc/hive/conf
spark.driver.extraJavaOptions -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hoodie-heapdump.hprof
spark.driver.maxResultSize 2g
spark.driver.memory 4g
@@ -551,9 +554,9 @@
spark.executor.id driver
spark.executor.instances 300
spark.executor.memory 6g
- spark.rdd.compress true</p>
+ spark.rdd.compress true
-<p>spark.kryoserializer.buffer.max 512m
+ spark.kryoserializer.buffer.max 512m
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.shuffle.memoryFraction 0.2
spark.shuffle.service.enabled true
@@ -561,12 +564,14 @@
spark.storage.memoryFraction 0.6
spark.submit.deployMode cluster
spark.task.cpus 1
- spark.task.maxFailures 4</p>
+ spark.task.maxFailures 4
-<p>spark.yarn.driver.memoryOverhead 1024
+ spark.yarn.driver.memoryOverhead 1024
spark.yarn.executor.memoryOverhead 3072
spark.yarn.max.executor.failures 100
- ```</p>
+
+</code></pre>
+</div>
<div class="tags">
@@ -582,9 +587,19 @@
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/dev_setup.html b/content/contributing.html
similarity index 52%
rename from content/dev_setup.html
rename to content/contributing.html
index 5f8a314..1901952 100644
--- a/content/dev_setup.html
+++ b/content/contributing.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li class="dropdownActive"><a href="dev_setup.html">Setup</a></li>
+ <li class="dropdownActive"><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -336,39 +337,116 @@
+
+
-<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
-<script>
-$( document ).ready(function() {
- // Handler for .ready() called.
-$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
+ <h2 id="pre-requisites">Pre-requisites</h2>
-/* this offset helps account for the space taken up by the floating toolbar. */
-$('#toc').on('click', 'a', function() {
- var target = $(this.getAttribute('href'))
- , scroll_target = target.offset().top
+<p>To contribute code, you need</p>
- $(window).scrollTop(scroll_target - 10);
- return false
-})
-
-});
-</script>
+<ul>
+ <li>a GitHub account</li>
+ <li>a Linux (or) macOS development environment with Java JDK 8, Apache Maven (3.x+) installed</li>
+ <li><a href="https://www.docker.com/">Docker</a> installed for running demo, integ tests or building website</li>
+ <li>for large contributions, a signed <a href="https://www.apache.org/licenses/icla.pdf">Individual Contributor License
+Agreement</a> (ICLA) to the Apache
+Software Foundation (ASF).</li>
+ <li>(Recommended) Create an account on <a href="https://issues.apache.org/jira/projects/HUDI/summary">JIRA</a> to open issues/find similar issues.</li>
+ <li>(Recommended) Join our dev mailing list & slack channel, listed on <a href="community.html">community</a> page.</li>
+</ul>
+
+<h2 id="ide-setup">IDE Setup</h2>
+
+<p>To contribute, you would need to fork the Hudi code on Github & then clone your own fork locally. Once cloned, we recommend building as per instructions on <a href="quickstart.html">quickstart</a></p>
+
+<p>We have embraced the code style largely based on <a href="https://google.github.io/styleguide/javaguide.html">google format</a>. Please setup your IDE with style files from <a href="../style/">here</a>.
+These instructions have been tested on IntelliJ. We also recommend setting up the <a href="https://plugins.jetbrains.com/plugin/7642-save-actions">Save Action Plugin</a> to auto format & organize imports on save. The Maven Compilation life-cycle will fail if there are checkstyle violations.</p>
+
+<h2 id="lifecycle">Lifecycle</h2>
+
+<p>Here’s a typical lifecycle of events to contribute to Hudi.</p>
+
+<ul>
+ <li>[Recommended] Share your intent on the mailing list, so that community can provide early feedback, point out any similar JIRAs or HIPs.</li>
+ <li>[Optional] If you want to get involved, but don’t have a project in mind, please check JIRA for small, quick-starters.</li>
+ <li>[Optional] Familiarize yourself with internals of Hudi using content on this page, as well as <a href="https://cwiki.apache.org/confluence/display/HUDI">wiki</a></li>
+ <li>Once you finalize on a project/task, please open a new JIRA or assign an existing one to yourself. (If you don’t have perms to do this, please email the dev mailing list with your JIRA id and a small intro for yourself. We’d be happy to add you as a contributor)</li>
+ <li>Make your code change
+ <ul>
+ <li>Every source file needs to include the Apache license header. Every new dependency needs to
+have an open source license <a href="https://www.apache.org/legal/resolved.html#criteria">compatible</a> with Apache.</li>
+ <li>Get existing tests to pass using <code class="highlighter-rouge">mvn clean install -DskipITs</code></li>
+ <li>Add adequate tests for your new functionality</li>
+ <li>[Optional] For involved changes, its best to also run the entire integration test suite using <code class="highlighter-rouge">mvn clean install</code></li>
+ <li>For website changes, please build the site locally & test navigation, formatting & links thoroughly</li>
+ </ul>
+ </li>
+ <li>Format commit messages and the pull request title like <code class="highlighter-rouge">[HUDI-XXX] Fixes bug in Spark Datasource</code>,
+where you replace HUDI-XXX with the appropriate JIRA issue.</li>
+ <li>Push your commit to your own fork/branch & create a pull request (PR) against the Hudi repo.</li>
+ <li>If you don’t hear back within 3 days on the PR, please send an email to dev @ mailing list.</li>
+ <li>Address code review comments & keep pushing changes to your fork/branch, which automatically updates the PR</li>
+ <li>Before your change can be merged, it should be squashed into a single commit for cleaner commit history.</li>
+</ul>
-<div id="toc"></div>
+<h2 id="releases">Releases</h2>
-
+<ul>
+ <li>Apache Hudi community plans to do minor version releases every 6 weeks or so.</li>
+ <li>If your contribution merged onto <code class="highlighter-rouge">master</code> branch after the last release, it will become part of next release.</li>
+ <li>Website changes are regenerated once a week (until automation in place to reflect immediately)</li>
+</ul>
+<h2 id="accounts-and-permissions">Accounts and Permissions</h2>
+
+<ul>
+ <li>
+ <p><a href="https://issues.apache.org/jira/projects/HUDI/issues">Hudi issue tracker (JIRA)</a>:
+Anyone can access it and browse issues. Anyone can register an account and login
+to create issues or add comments. Only contributors can be assigned issues. If
+you want to be assigned issues, a PMC member can add you to the project contributor
+group. Email the dev mailing list to ask to be added as a contributor, and include your ASF Jira username.</p>
+ </li>
+ <li>
+ <p><a href="https://cwiki.apache.org/confluence/display/HUDI">Hudi Wiki Space</a>:
+Anyone has read access. If you wish to contribute changes, please create an account and
+request edit access on the dev@ mailing list (include your Wiki account user ID).</p>
+ </li>
+ <li>
+ <p>Pull requests can only be merged by a HUDI committer, listed <a href="https://incubator.apache.org/projects/hudi.html">here</a></p>
+ </li>
+ <li>
+ <p><a href="https://www.apache.org/foundation/voting.html">Voting on a release</a>: Everyone can vote.
+Only Hudi PMC members should mark their votes as binding.</p>
+ </li>
+</ul>
-
+<h2 id="communication">Communication</h2>
+
+<p>All communication is expected to align with the <a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a>.
+Discussion about contributing code to Hudi happens on the <a href="community.html">dev@ mailing list</a>. Introduce yourself!</p>
+
+<h2 id="code--project-structure">Code & Project Structure</h2>
+
+<ul>
+ <li><code class="highlighter-rouge">docker</code> : Docker containers used by demo and integration tests. Brings up a mini data ecosystem locally</li>
+ <li><code class="highlighter-rouge">hoodie-cli</code> : CLI to inspect, manage and administer datasets</li>
+ <li><code class="highlighter-rouge">hoodie-client</code> : Spark client library to take a bunch of inserts + updates and apply them to a Hoodie table</li>
+ <li><code class="highlighter-rouge">hoodie-common</code> : Common classes used across modules</li>
+ <li><code class="highlighter-rouge">hoodie-hadoop-mr</code> : InputFormat implementations for ReadOptimized, Incremental, Realtime views</li>
+ <li><code class="highlighter-rouge">hoodie-hive</code> : Manage hive tables off Hudi datasets and houses the HiveSyncTool</li>
+ <li><code class="highlighter-rouge">hoodie-integ-test</code> : Longer running integration test processes</li>
+ <li><code class="highlighter-rouge">hoodie-spark</code> : Spark datasource for writing and reading Hudi datasets. Streaming sink.</li>
+ <li><code class="highlighter-rouge">hoodie-utilities</code> : Houses tools like DeltaStreamer, SnapshotCopier</li>
+ <li><code class="highlighter-rouge">packaging</code> : Poms for building out bundles for easier drop in to Spark, Hive, Presto</li>
+ <li><code class="highlighter-rouge">style</code> : Code formatting, checkstyle files</li>
+</ul>
- <h3 id="code-style">Code Style</h3>
+<h2 id="website">Website</h2>
-<p>We have embraced the code style largely based on <a href="https://google.github.io/styleguide/javaguide.html">google format</a>.
- Please setup your IDE with style files from <a href="../style/">here</a>
- We also recommend setting up the <a href="https://plugins.jetbrains.com/plugin/7642-save-actions">Save Action Plugin</a> to auto format & organize imports on save.
- The Maven Compilation life-cycle will fail if there are checkstyle violations.</p>
+<p><a href="https://hudi.apache.org">Apache Hudi site</a> is hosted on a special <code class="highlighter-rouge">asf-site</code> branch. Please follow the <code class="highlighter-rouge">README</code> file under <code class="highlighter-rouge">docs</code> on that branch for
+instructions on making changes to the website.</p>
<div class="tags">
@@ -384,9 +462,19 @@ $('#toc').on('click', 'a', function() {
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/css/customstyles.css b/content/css/customstyles.css
index 676c804..d6667a5 100644
--- a/content/css/customstyles.css
+++ b/content/css/customstyles.css
@@ -83,7 +83,7 @@ padding-top: 60px;
margin-top: -40px
}
-body h1 {margin-top:40px;}
+body h1 {margin-top:80px;}
.post-content img {
margin: 12px 0px 3px 0px;
@@ -1178,4 +1178,4 @@ div.box.box1 {
h4.panel-title {
padding-top: 0px;
margin-top: 0px;
-}
\ No newline at end of file
+}
diff --git a/content/css/lavish-bootstrap.css b/content/css/lavish-bootstrap.css
index ff64532..a050c9a 100644
--- a/content/css/lavish-bootstrap.css
+++ b/content/css/lavish-bootstrap.css
@@ -3643,6 +3643,14 @@ textarea.input-group-sm > .input-group-btn > .btn {
.navbar-brand:focus {
text-decoration: none;
}
+.navbar-incubate {
+ text-align: center;
+ font-size: 24px;
+ color: grey;
+}
+
+
+
@media (min-width: 768px) {
.navbar > .container .navbar-brand {
margin-left: -15px;
diff --git a/content/css/theme-blue.css b/content/css/theme-blue.css
index f9aa248..9a923ef 100644
--- a/content/css/theme-blue.css
+++ b/content/css/theme-blue.css
@@ -95,11 +95,10 @@ div.sidebarTitle {
}
li.sidebarTitle {
- margin-top:40px;
+ margin-top:80px;
font-weight:normal;
font-size:130%;
color: #333;
margin-bottom:10px;
margin-left: 5px;
-
}
diff --git a/content/feed.xml b/content/feed.xml
index 3a620b8..b21704e 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -2,14 +2,28 @@
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title></title>
- <description>Apache Hudi (pronounced “Hoodie”) is a Spark Library, that provides upserts and incremental processing capaibilities on Hadoop datasets</description>
+ <description>Apache Hudi (pronounced “Hoodie”) provides upserts and incremental processing capaibilities on Big Data</description>
<link>http://0.0.0.0:4000/</link>
<atom:link href="http://0.0.0.0:4000/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Sat, 16 Feb 2019 04:53:14 +0000</pubDate>
- <lastBuildDate>Sat, 16 Feb 2019 04:53:14 +0000</lastBuildDate>
+ <pubDate>Mon, 25 Feb 2019 20:49:33 +0000</pubDate>
+ <lastBuildDate>Mon, 25 Feb 2019 20:49:33 +0000</lastBuildDate>
<generator>Jekyll v3.3.1</generator>
<item>
+ <title>Hudi entered Apache Incubator</title>
+ <description><p>In the coming weeks, we will be moving in our new home on the Apache Incubator.</p>
+
+</description>
+ <pubDate>Fri, 18 Jan 2019 00:00:00 +0000</pubDate>
+ <link>http://0.0.0.0:4000/strata-talk.html</link>
+ <guid isPermaLink="true">http://0.0.0.0:4000/strata-talk.html</guid>
+
+ <category>news</category>
+
+
+ </item>
+
+ <item>
<title>Connect with us at Strata San Jose March 2017</title>
<description><p>We will be presenting Hoodie &amp; general concepts around how incremental processing works at Uber.
Catch our talk <strong>“Incremental Processing on Hadoop At Uber”</strong></p>
diff --git a/content/gcs_hoodie.html b/content/gcs_hoodie.html
index 847b72b..f90992d 100644
--- a/content/gcs_hoodie.html
+++ b/content/gcs_hoodie.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -415,9 +416,19 @@
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/images/asf_logo.svg b/content/images/asf_logo.svg
new file mode 100644
index 0000000..620694c
--- /dev/null
+++ b/content/images/asf_logo.svg
@@ -0,0 +1,210 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 19.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg version="1.1" id="Layer_2" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+ viewBox="0 0 7127.6 2890" enable-background="new 0 0 7127.6 2890" xml:space="preserve">
+<path fill="#6D6E71" d="M7104.7,847.8c15.3,15.3,22.9,33.7,22.9,55.2c0,21.5-7.6,39.9-22.9,55.4c-15.3,15.4-33.8,23.1-55.6,23.1
+ c-21.8,0-40.2-7.6-55.4-22.9c-15.1-15.3-22.7-33.7-22.7-55.2c0-21.5,7.6-39.9,22.9-55.4c15.3-15.4,33.7-23.1,55.4-23.1
+ C7070.9,824.9,7089.4,832.5,7104.7,847.8z M7098.1,951.9c13.3-13.6,20-29.8,20-48.7s-6.6-35-19.8-48.5
+ c-13.2-13.4-29.4-20.1-48.6-20.1c-19.2,0-35.4,6.7-48.7,20.2c-13.3,13.5-19.9,29.7-19.9,48.7c0,19,6.6,35.2,19.7,48.6
+ c13.1,13.4,29.3,20.1,48.5,20.1S7084.7,965.4,7098.1,951.9z M7087.1,888.1c0,14-6.1,22.8-18.4,26.4l22.5,30.5h-18.2l-20.3-28.3
+ h-18.6v28.3h-14.7v-84.6h31.8c12.8,0,22,2.2,27.6,6.6C7084.4,871.4,7087.1,878.4,7087.1,888.1z M7068.2,900c3-2.4,4.4-6.5,4.4-12
+ c0-5.5-1.5-9.4-4.5-11.6c-3-2.2-8.4-3.2-16-3.2h-18v30.5h17.5C7059.7,903.6,7065.3,902.4,7068.2,900z"/>
+<path fill="#6D6E71" d="M1803.6,499.8v155.4h-20V499.8h-56.8v-19.2h133.9v19.2H1803.6z"/>
+<path fill="#6D6E71" d="M2082.2,655.2v-76.9h-105.2v76.9h-20V480.5h20v78.9h105.2v-78.9h20v174.7H2082.2z"/>
+<path fill="#6D6E71" d="M2241.4,499.8v57.4h88.1v19.2h-88.1v59.8h101.8v19h-121.8V480.5H2340v19.2H2241.4z"/>
+<path fill="#D22128" d="M1574.5,1852.4l417.3-997.6h80.1l417.3,997.6h-105.4l-129.3-311.9h-448.2l-127.9,311.9H1574.5z M2032.6,970
+ l-205.1,493.2h404.7L2032.6,970z"/>
+<path fill="#D22128" d="M2596.9,1852.4V854.8H3010c171.4,0,295.1,158.8,295.1,313.3c0,163-115.2,316.1-286.6,316.1h-324.6v368.1
+ H2596.9z M2693.9,1397.1h318.9c118,0,193.9-108.2,193.9-229c0-125.1-92.7-226.2-202.3-226.2h-310.5V1397.1z"/>
+<path fill="#D22128" d="M3250.5,1852.4l417.3-997.6h80.1l417.3,997.6h-105.4l-129.3-311.9h-448.2l-127.9,311.9H3250.5z M3708.6,970
+ l-205.1,493.2h404.7L3708.6,970z"/>
+<path fill="#D22128" d="M4637.3,849.1c177,0,306.3,89.9,368.1,217.8l-78.7,47.8c-63.2-132.1-186.9-177-295.1-177
+ c-238.9,0-369.5,213.6-369.5,414.5c0,220.6,161.6,420.1,373.7,420.1c112.4,0,244.5-56.2,307.7-185.5l81.5,42.1
+ c-64.6,148.9-241.7,231.8-394.8,231.8c-274,0-466.5-261.3-466.5-514.2C4163.8,1106.3,4336.6,849.1,4637.3,849.1z"/>
+<path fill="#D22128" d="M5949.1,854.8v997.6h-98.4v-466.5h-591.5v466.5h-96.9V854.8h96.9v444h591.5v-444H5949.1z"/>
+<path fill="#D22128" d="M6844.6,1765.2v87.1h-670.2V854.8H6832v87.1h-560.6v359.7h489v82.9h-489v380.8H6844.6z"/>
+<path fill="#6D6E71" d="M1667.6,2063.6c11.8,3.5,22.2,8.3,31,14.2l-10.3,22.6c-9-6-18.6-10.4-28.9-13.4c-10.2-2.9-20-4.4-29.2-4.4
+ c-13.6,0-24.5,2.4-32.6,7.3c-8.1,4.9-12.2,11.8-12.2,20.7c0,7.6,2.3,14,6.8,19c4.5,5,10.2,8.9,17,11.7c6.8,2.8,16.1,6,28,9.6
+ c14.4,4.6,26,8.9,34.7,12.9c8.8,4,16.3,9.9,22.5,17.8c6.2,7.8,9.3,18.2,9.3,31c0,11.7-3.2,21.8-9.5,30.6
+ c-6.3,8.7-15.3,15.5-26.8,20.3c-11.6,4.8-24.9,7.2-40,7.2c-15.1,0-29.7-2.9-43.9-8.7c-14.2-5.8-26.4-13.6-36.6-23.4l10.7-21.6
+ c9.6,9.4,20.7,16.7,33.3,21.9c12.6,5.2,24.8,7.8,36.8,7.8c15.3,0,27.3-3,36.1-8.9c8.8-5.9,13.2-13.9,13.2-23.9
+ c0-7.8-2.3-14.3-6.9-19.4c-4.6-5.1-10.3-9-17.1-11.9c-6.8-2.8-16.1-6-28-9.6c-14.2-4.2-25.7-8.3-34.6-12.2
+ c-8.9-3.9-16.4-9.7-22.5-17.5c-6.1-7.7-9.2-17.9-9.2-30.6c0-10.9,3-20.4,9-28.6c6-8.2,14.6-14.6,25.6-19.1
+ c11.1-4.5,23.8-6.8,38.2-6.8C1643.8,2058.3,1655.7,2060.1,1667.6,2063.6z"/>
+<path fill="#6D6E71" d="M1980.1,2072.8c16.8,9.4,30.2,22.3,40,38.4c9.8,16.2,14.8,33.9,14.8,53.3c0,19.5-4.9,37.4-14.8,53.6
+ c-9.8,16.3-23.2,29.1-40,38.6c-16.8,9.5-35.3,14.3-55.2,14.3c-20.3,0-38.8-4.7-55.7-14.3c-16.8-9.5-30.2-22.4-40-38.6
+ c-9.8-16.3-14.8-34.1-14.8-53.6c0-19.5,4.9-37.3,14.8-53.5c9.8-16.2,23.2-29,40-38.3c16.8-9.4,35.4-14,55.7-14
+ C1944.8,2058.6,1963.2,2063.3,1980.1,2072.8z M1881.9,2092.7c-13.1,7.4-23.6,17.5-31.4,30.1c-7.8,12.6-11.8,26.5-11.8,41.7
+ c0,15.3,3.9,29.3,11.8,42c7.8,12.7,18.3,22.8,31.4,30.2c13.1,7.4,27.4,11.1,42.9,11.1c15.5,0,29.7-3.7,42.7-11.1
+ c13-7.4,23.3-17.4,31.1-30.2c7.7-12.7,11.6-26.7,11.6-42s-3.9-29.2-11.6-41.8c-7.7-12.6-18.1-22.6-31.1-30
+ c-13-7.4-27.2-11.2-42.6-11.2C1909.4,2081.5,1895.1,2085.2,1881.9,2092.7z"/>
+<path fill="#6D6E71" d="M2186.5,2082.4v74h98.4v23.2h-98.4v90.2h-24.1v-210.6h133.8v23.2H2186.5z"/>
+<path fill="#6D6E71" d="M2491.6,2082.4v187.4h-24.1v-187.4h-68.4v-23.2h161.4v23.2H2491.6z"/>
+<path fill="#6D6E71" d="M2871.8,2269.8l-56.8-177.4l-57.6,177.4h-24.5l-70.5-210.6h25.9l57.9,182.7l57.1-182.4l24.1-0.3l57.7,182.7
+ l57.1-182.7h25l-70.6,210.6H2871.8z"/>
+<path fill="#6D6E71" d="M3087.3,2216.6l-23.5,53.2h-25.6l94.4-210.6h25l94.1,210.6h-26.1l-23.5-53.2H3087.3z M3144.5,2086.6
+ l-46.9,106.8h94.4L3144.5,2086.6z"/>
+<path fill="#6D6E71" d="M3461.1,2202.7c-6,0.4-10.7,0.6-14.1,0.6h-56v66.5H3367v-210.6h80c26.2,0,46.6,6.2,61.2,18.5
+ c14.5,12.3,21.8,29.8,21.8,52.3c0,17.2-4.1,31.7-12.2,43.3c-8.1,11.6-19.8,20-35,25l49.2,71.5h-27.3L3461.1,2202.7z M3491.3,2167.6
+ c10.3-8.4,15.5-20.8,15.5-37c0-15.9-5.2-27.9-15.5-36c-10.3-8.1-25.1-12.2-44.3-12.2h-56v97.8h56
+ C3466.2,2180.2,3481,2176,3491.3,2167.6z"/>
+<path fill="#6D6E71" d="M3688.3,2082.4v69.2h106.2v23.2h-106.2v72.1h122.8v22.9h-146.9v-210.6h142.9v23.2H3688.3z"/>
+<path fill="#6D6E71" d="M4147,2082.4v74h98.4v23.2H4147v90.2h-24.1v-210.6h133.8v23.2H4147z"/>
+<path fill="#6D6E71" d="M4523.3,2072.8c16.8,9.4,30.2,22.3,40,38.4c9.8,16.2,14.8,33.9,14.8,53.3c0,19.5-4.9,37.4-14.8,53.6
+ c-9.8,16.3-23.2,29.1-40,38.6c-16.8,9.5-35.3,14.3-55.2,14.3c-20.3,0-38.8-4.7-55.7-14.3c-16.8-9.5-30.2-22.4-40-38.6
+ c-9.8-16.3-14.8-34.1-14.8-53.6c0-19.5,4.9-37.3,14.8-53.5c9.8-16.2,23.2-29,40-38.3c16.8-9.4,35.4-14,55.7-14
+ C4488.1,2058.6,4506.5,2063.3,4523.3,2072.8z M4425.2,2092.7c-13.1,7.4-23.6,17.5-31.4,30.1c-7.8,12.6-11.8,26.5-11.8,41.7
+ c0,15.3,3.9,29.3,11.8,42c7.8,12.7,18.3,22.8,31.4,30.2c13.1,7.4,27.4,11.1,42.9,11.1c15.5,0,29.7-3.7,42.7-11.1
+ c13-7.4,23.3-17.4,31.1-30.2c7.7-12.7,11.6-26.7,11.6-42s-3.9-29.2-11.6-41.8c-7.7-12.6-18.1-22.6-31.1-30
+ c-13-7.4-27.2-11.2-42.6-11.2C4452.6,2081.5,4438.3,2085.2,4425.2,2092.7z"/>
+<path fill="#6D6E71" d="M4854.7,2247.7c-15.7,15.5-37.3,23.3-64.8,23.3c-27.7,0-49.4-7.8-65.1-23.3c-15.7-15.5-23.6-37-23.6-64.6
+ v-124h24.1v124c0,20.3,5.8,36.1,17.3,47.5c11.6,11.4,27.3,17.1,47.3,17.1c20.1,0,35.8-5.7,47.1-17c11.4-11.3,17-27.2,17-47.7v-124
+ h24.1v124C4878.2,2210.7,4870.4,2232.2,4854.7,2247.7z"/>
+<path fill="#6D6E71" d="M5169.5,2269.8l-126.3-169.1v169.1h-24.1v-210.6h25l126.3,169.3v-169.3h23.8v210.6H5169.5z"/>
+<path fill="#6D6E71" d="M5478.4,2073.1c16.4,9.3,29.4,21.9,38.9,37.9c9.6,16,14.3,33.9,14.3,53.5s-4.8,37.6-14.3,53.6
+ c-9.5,16.1-22.6,28.7-39.3,37.9c-16.6,9.2-35.2,13.8-55.5,13.8h-84.3v-210.6h85.2C5443.7,2059.2,5462,2063.8,5478.4,2073.1z
+ M5362.3,2246.9h61.4c15.5,0,29.6-3.5,42.3-10.6c12.7-7.1,22.8-16.9,30.2-29.5c7.4-12.5,11.1-26.5,11.1-42
+ c0-15.5-3.8-29.4-11.3-41.9c-7.5-12.5-17.7-22.3-30.6-29.6c-12.8-7.2-27-10.9-42.6-10.9h-60.5V2246.9z"/>
+<path fill="#6D6E71" d="M5668.6,2216.6l-23.5,53.2h-25.6l94.4-210.6h25l94.1,210.6H5807l-23.5-53.2H5668.6z M5725.8,2086.6
+ l-46.9,106.8h94.4L5725.8,2086.6z"/>
+<path fill="#6D6E71" d="M5991,2082.4v187.4H5967v-187.4h-68.4v-23.2h161.4v23.2H5991z"/>
+<path fill="#6D6E71" d="M6175.9,2269.8v-210.6h24.1v210.6H6175.9z"/>
+<path fill="#6D6E71" d="M6493.7,2072.8c16.8,9.4,30.2,22.3,40,38.4c9.8,16.2,14.8,33.9,14.8,53.3c0,19.5-4.9,37.4-14.8,53.6
+ c-9.8,16.3-23.2,29.1-40,38.6c-16.8,9.5-35.3,14.3-55.2,14.3c-20.3,0-38.8-4.7-55.7-14.3c-16.8-9.5-30.2-22.4-40-38.6
+ c-9.8-16.3-14.8-34.1-14.8-53.6c0-19.5,4.9-37.3,14.8-53.5c9.8-16.2,23.2-29,40-38.3c16.8-9.4,35.4-14,55.7-14
+ C6458.5,2058.6,6476.9,2063.3,6493.7,2072.8z M6395.6,2092.7c-13.1,7.4-23.6,17.5-31.4,30.1c-7.8,12.6-11.8,26.5-11.8,41.7
+ c0,15.3,3.9,29.3,11.8,42c7.8,12.7,18.3,22.8,31.4,30.2c13.1,7.4,27.4,11.1,42.9,11.1c15.5,0,29.7-3.7,42.7-11.1
+ c13-7.4,23.3-17.4,31.1-30.2c7.7-12.7,11.6-26.7,11.6-42s-3.9-29.2-11.6-41.8c-7.7-12.6-18.1-22.6-31.1-30
+ c-13-7.4-27.2-11.2-42.6-11.2C6423,2081.5,6408.8,2085.2,6395.6,2092.7z"/>
+<path fill="#6D6E71" d="M6826.5,2269.8l-126.3-169.1v169.1h-24.1v-210.6h25l126.3,169.3v-169.3h23.8v210.6H6826.5z"/>
+<linearGradient id="SVGID_1_" gradientUnits="userSpaceOnUse" x1="-4516.6152" y1="-2338.7222" x2="-4108.4111" y2="-1861.3982" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0" style="stop-color:#F69923"/>
+ <stop offset="0.3123" style="stop-color:#F79A23"/>
+ <stop offset="0.8383" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_1_)" d="M1230.1,13.7c-45.3,26.8-120.6,102.5-210.5,212.3l82.6,155.9c58-82.9,116.9-157.5,176.3-221.2
+ c4.6-5.1,7-7.5,7-7.5c-2.3,2.5-4.6,5-7,7.5c-19.2,21.2-77.5,89.2-165.5,224.4c84.7-4.2,214.9-21.6,321.1-39.7
+ c31.6-177-31-258-31-258S1323.4-41.4,1230.1,13.7z"/>
+<path fill="none" d="M1090.2,903.1c0.6-0.1,1.2-0.2,1.8-0.3l-11.9,1.3c-0.7,0.3-1.4,0.7-2.1,1
+ C1082.1,904.4,1086.2,903.7,1090.2,903.1z"/>
+<path fill="none" d="M1005.9,1182.3c-6.7,1.5-13.7,2.7-20.7,3.7C992.3,1185,999.2,1183.8,1005.9,1182.3z"/>
+<path fill="none" d="M432.9,1808.8c0.9-2.3,1.8-4.7,2.6-7c18.2-48,36.2-94.7,54-140.1c20-51,39.8-100.4,59.3-148.3
+ c20.6-50.4,40.9-99.2,60.9-146.3c21-49.4,41.7-97,62-142.8c16.5-37.3,32.8-73.4,48.9-108.3c5.4-11.7,10.7-23.2,16-34.6
+ c10.5-22.7,21-44.8,31.3-66.5c9.5-20,19-39.6,28.3-58.8c3.1-6.4,6.2-12.8,9.3-19.1c0.5-1,1-2,1.5-3.1l-10.2,1.1l-8-15.9
+ c-0.8,1.6-1.6,3.1-2.4,4.6c-14.5,28.8-28.9,57.9-43.1,87.2c-8.2,16.9-16.4,34-24.6,51c-22.6,47.4-44.8,95.2-66.6,143.3
+ c-22.1,48.6-43.7,97.5-64.9,146.5c-20.8,48.1-41.3,96.2-61.2,144.2c-20,48-39.5,95.7-58.5,143.2c-19.9,49.5-39.2,98.7-58,147.2
+ c-4.2,10.9-8.5,21.9-12.7,32.8c-15,39.2-29.7,77.8-44,116l12.7,25.1l11.4-1.2c0.4-1.1,0.8-2.3,1.3-3.4
+ C396.7,1905.4,414.9,1856.4,432.9,1808.8z"/>
+<path fill="none" d="M980,1186.8L980,1186.8c0.1,0,0.1,0,0.1-0.1C980.1,1186.8,980.1,1186.8,980,1186.8z"/>
+<path fill="#BE202E" d="M952.6,1323c-10.6,1.9-21.4,3.8-32.5,5.7c-0.1,0-0.1,0.1-0.2,0.1c5.6-0.8,11.2-1.7,16.6-2.6
+ C942,1325.2,947.3,1324.1,952.6,1323z"/>
+<path opacity="0.35" fill="#BE202E" d="M952.6,1323c-10.6,1.9-21.4,3.8-32.5,5.7c-0.1,0-0.1,0.1-0.2,0.1c5.6-0.8,11.2-1.7,16.6-2.6
+ C942,1325.2,947.3,1324.1,952.6,1323z"/>
+<path fill="#BE202E" d="M980.3,1186.7C980.2,1186.7,980.2,1186.7,980.3,1186.7c-0.1,0.1-0.2,0.1-0.2,0.1c1.8-0.2,3.5-0.5,5.2-0.8
+ c7-1,13.9-2.2,20.7-3.7C997.5,1183.8,989,1185.2,980.3,1186.7L980.3,1186.7L980.3,1186.7z"/>
+<path opacity="0.35" fill="#BE202E" d="M980.3,1186.7C980.2,1186.7,980.2,1186.7,980.3,1186.7c-0.1,0.1-0.2,0.1-0.2,0.1
+ c1.8-0.2,3.5-0.5,5.2-0.8c7-1,13.9-2.2,20.7-3.7C997.5,1183.8,989,1185.2,980.3,1186.7L980.3,1186.7L980.3,1186.7z"/>
+<linearGradient id="SVGID_2_" gradientUnits="userSpaceOnUse" x1="-7537.7339" y1="-2391.4075" x2="-4625.4141" y2="-2391.4075" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_2_)" d="M858.6,784.7c25.1-46.9,50.5-92.8,76.2-137.4c26.7-46.4,53.7-91.3,80.9-134.7
+ c1.6-2.6,3.2-5.2,4.8-7.7c27-42.7,54.2-83.7,81.6-122.9L1019.5,226c-6.2,7.6-12.5,15.3-18.8,23.2c-23.8,29.7-48.6,61.6-73.9,95.5
+ c-28.6,38.2-58,78.9-87.8,121.7c-27.6,39.5-55.5,80.9-83.5,123.7c-23.8,36.5-47.7,74-71.4,112.5c-0.9,1.4-1.8,2.9-2.6,4.3
+ l107.5,212.3C811.8,873.6,835.1,828.7,858.6,784.7z"/>
+<linearGradient id="SVGID_3_" gradientUnits="userSpaceOnUse" x1="-7186.1777" y1="-2099.3059" x2="-5450.7183" y2="-2099.3059" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0" style="stop-color:#282662"/>
+ <stop offset="9.548390e-02" style="stop-color:#662E8D"/>
+ <stop offset="0.7882" style="stop-color:#9F2064"/>
+ <stop offset="0.9487" style="stop-color:#CD2032"/>
+</linearGradient>
+<path fill="url(#SVGID_3_)" d="M369,1981c-14.2,39.1-28.5,78.9-42.9,119.6c-0.2,0.6-0.4,1.2-0.6,1.8c-2,5.7-4.1,11.5-6.1,17.2
+ c-9.7,27.4-18,52.1-37.3,108.2c31.7,14.5,57.1,52.5,81.1,95.6c-2.6-44.7-21-86.6-56.2-119.1c156.1,7,290.6-32.4,360.1-146.6
+ c6.2-10.2,11.9-20.9,17-32.2c-31.6,40.1-70.8,57.1-144.5,53c-0.2,0.1-0.3,0.1-0.5,0.2c0.2-0.1,0.3-0.1,0.5-0.2
+ c108.6-48.6,163.1-95.3,211.2-172.6c11.4-18.3,22.5-38.4,33.8-60.6c-94.9,97.5-205,125.3-320.9,104.2l-86.9,9.5
+ C374.4,1966.3,371.7,1973.6,369,1981z"/>
+<linearGradient id="SVGID_4_" gradientUnits="userSpaceOnUse" x1="-7374.1626" y1="-2418.5454" x2="-4461.8428" y2="-2418.5454" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_4_)" d="M409.6,1786.3c18.8-48.5,38.1-97.7,58-147.2c19-47.4,38.5-95.2,58.5-143.2
+ c20-48,40.4-96.1,61.2-144.2c21.2-49,42.9-97.8,64.9-146.5c21.8-48.1,44-95.9,66.6-143.3c8.1-17.1,16.3-34.1,24.6-51
+ c14.2-29.3,28.6-58.4,43.1-87.2c0.8-1.6,1.6-3.1,2.4-4.6L681.4,706.8c-1.8,2.9-3.5,5.8-5.3,8.6c-25.1,40.9-50,82.7-74.4,125.4
+ c-24.7,43.1-49,87.1-72.7,131.7c-20,37.6-39.6,75.6-58.6,113.9c-3.8,7.8-7.6,15.5-11.3,23.2c-23.4,48.2-44.6,94.8-63.7,139.5
+ c-21.7,50.7-40.7,99.2-57.5,145.1c-11,30.2-21,59.4-30.1,87.4c-7.5,24-14.7,47.9-21.5,71.8c-16,56.3-29.9,112.4-41.2,168.3
+ L353,1935.1c14.3-38.1,28.9-76.8,44-116C401.1,1808.2,405.4,1797.3,409.6,1786.3z"/>
+<linearGradient id="SVGID_5_" gradientUnits="userSpaceOnUse" x1="-7161.7642" y1="-2379.1431" x2="-5631.2524" y2="-2379.1431" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0" style="stop-color:#282662"/>
+ <stop offset="9.548390e-02" style="stop-color:#662E8D"/>
+ <stop offset="0.7882" style="stop-color:#9F2064"/>
+ <stop offset="0.9487" style="stop-color:#CD2032"/>
+</linearGradient>
+<path fill="url(#SVGID_5_)" d="M243.5,1729.4c-13.6,68.2-23.2,136.2-28,203.8c-0.2,2.4-0.4,4.7-0.5,7.1
+ c-33.7-54-124-106.8-123.8-106.2c64.6,93.7,113.7,186.7,120.9,278c-34.6,7.1-82-3.2-136.8-23.3c57.1,52.5,100,67,116.7,70.9
+ c-52.5,3.3-107.1,39.3-162.1,80.8c80.5-32.8,145.5-45.8,192.1-35.3C148.1,2414.2,74.1,2645,0,2890c22.7-6.7,36.2-21.9,43.9-42.6
+ c13.2-44.4,100.8-335.6,238-718.2c3.9-10.9,7.8-21.8,11.8-32.9c1.1-3,2.2-6.1,3.3-9.2c14.5-40.1,29.5-81.1,45.1-122.9
+ c3.5-9.5,7.1-19,10.7-28.6c0.1-0.2,0.1-0.4,0.2-0.6l-107.9-213.2C244.6,1724.4,244,1726.9,243.5,1729.4z"/>
+<linearGradient id="SVGID_6_" gradientUnits="userSpaceOnUse" x1="-7374.1626" y1="-2117.1309" x2="-4461.8428" y2="-2117.1309" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_6_)" d="M805.6,937c-3.1,6.3-6.2,12.7-9.3,19.1c-9.3,19.2-18.8,38.8-28.3,58.8
+ c-10.3,21.7-20.7,43.9-31.3,66.5c-5.3,11.4-10.6,22.9-16,34.6c-16.1,35-32.4,71.1-48.9,108.3c-20.3,45.8-41,93.4-62,142.8
+ c-20,47.1-40.3,95.9-60.9,146.3c-19.5,47.9-39.3,97.3-59.3,148.3c-17.8,45.4-35.9,92.1-54,140.1c-0.9,2.3-1.8,4.7-2.6,7
+ c-18,47.6-36.2,96.6-54.6,146.8c-0.4,1.1-0.8,2.3-1.3,3.4l86.9-9.5c-1.7-0.3-3.5-0.5-5.2-0.9c103.9-13,242.1-90.6,331.4-186.5
+ c41.1-44.2,78.5-96.3,113-157.3c25.7-45.4,49.8-95.8,72.8-151.5c20.1-48.7,39.4-101.4,58-158.6c-23.9,12.6-51.2,21.8-81.4,28.2
+ c-5.3,1.1-10.7,2.2-16.1,3.1c-5.5,1-11,1.8-16.6,2.6l0,0l0,0c0.1,0,0.1-0.1,0.2-0.1c96.9-37.3,158-109.2,202.4-197.4
+ c-25.5,17.4-66.9,40.1-116.6,51.1c-6.7,1.5-13.7,2.7-20.7,3.7c-1.7,0.3-3.5,0.6-5.2,0.8l0,0l0,0c0.1,0,0.1,0,0.1-0.1
+ c0,0,0.1,0,0.1,0l0,0c33.6-14.1,62-29.8,86.6-48.4c5.3-4,10.4-8.1,15.3-12.3c7.5-6.5,14.7-13.3,21.5-20.5c4.4-4.6,8.6-9.3,12.7-14.2
+ c9.6-11.5,18.7-23.9,27.1-37.3c2.6-4.1,5.1-8.3,7.6-12.6c3.2-6.2,6.3-12.3,9.3-18.3c13.5-27.2,24.4-51.5,33-72.8
+ c4.3-10.6,8.1-20.5,11.3-29.7c1.3-3.7,2.5-7.2,3.7-10.6c3.4-10.2,6.2-19.3,8.4-27.3c3.3-12,5.3-21.5,6.4-28.4l0,0l0,0
+ c-3.3,2.6-7.1,5.2-11.3,7.7c-29.3,17.5-79.5,33.4-119.9,40.8l79.8-8.8l-79.8,8.8c-0.6,0.1-1.2,0.2-1.8,0.3c-4,0.7-8.1,1.3-12.2,2
+ c0.7-0.3,1.4-0.7,2.1-1l-273,29.9C806.6,935,806.1,936,805.6,937z"/>
+<linearGradient id="SVGID_7_" gradientUnits="userSpaceOnUse" x1="-7554.8232" y1="-2132.0981" x2="-4642.5034" y2="-2132.0981" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_7_)" d="M1112.9,385.1c-24.3,37.3-50.8,79.6-79.4,127.5c-1.5,2.5-3,5.1-4.5,7.6
+ c-24.6,41.5-50.8,87.1-78.3,137c-23.8,43.1-48.5,89.3-74.3,139c-22.4,43.3-45.6,89.2-69.4,137.8l273-29.9
+ c79.5-36.6,115.1-69.7,149.6-117.6c9.2-13.2,18.4-27,27.5-41.3c28-43.8,55.6-92,80.1-139.9c23.7-46.3,44.7-92.2,60.7-133.5
+ c10.2-26.3,18.4-50.8,24.1-72.3c5-19,8.9-36.9,11.9-54.1C1327.9,363.5,1197.6,380.9,1112.9,385.1z"/>
+<path fill="#BE202E" d="M936.5,1326.1c-5.5,1-11,1.8-16.6,2.6l0,0C925.5,1328,931,1327.1,936.5,1326.1z"/>
+<path opacity="0.35" fill="#BE202E" d="M936.5,1326.1c-5.5,1-11,1.8-16.6,2.6l0,0C925.5,1328,931,1327.1,936.5,1326.1z"/>
+<linearGradient id="SVGID_8_" gradientUnits="userSpaceOnUse" x1="-7374.1626" y1="-2027.484" x2="-4461.8433" y2="-2027.484" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_8_)" d="M936.5,1326.1c-5.5,1-11,1.8-16.6,2.6l0,0C925.5,1328,931,1327.1,936.5,1326.1z"/>
+<path fill="#BE202E" d="M980,1186.8c1.8-0.2,3.5-0.5,5.2-0.8C983.5,1186.3,981.8,1186.6,980,1186.8L980,1186.8z"/>
+<path opacity="0.35" fill="#BE202E" d="M980,1186.8c1.8-0.2,3.5-0.5,5.2-0.8C983.5,1186.3,981.8,1186.6,980,1186.8L980,1186.8z"/>
+<linearGradient id="SVGID_9_" gradientUnits="userSpaceOnUse" x1="-7374.1626" y1="-2037.7417" x2="-4461.8433" y2="-2037.7417" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_9_)" d="M980,1186.8c1.8-0.2,3.5-0.5,5.2-0.8C983.5,1186.3,981.8,1186.6,980,1186.8L980,1186.8z"/>
+<path fill="#BE202E" d="M980.2,1186.7C980.2,1186.7,980.2,1186.7,980.2,1186.7L980.2,1186.7L980.2,1186.7L980.2,1186.7
+ C980.2,1186.7,980.2,1186.7,980.2,1186.7z"/>
+<path opacity="0.35" fill="#BE202E" d="M980.2,1186.7C980.2,1186.7,980.2,1186.7,980.2,1186.7L980.2,1186.7L980.2,1186.7
+ L980.2,1186.7C980.2,1186.7,980.2,1186.7,980.2,1186.7z"/>
+<linearGradient id="SVGID_10_" gradientUnits="userSpaceOnUse" x1="-5738.0635" y1="-2039.799" x2="-5094.3457" y2="-2039.799" gradientTransform="matrix(0.4226 -0.9063 0.9063 0.4226 5117.8774 -2859.9343)">
+ <stop offset="0.3233" style="stop-color:#9E2064"/>
+ <stop offset="0.6302" style="stop-color:#C92037"/>
+ <stop offset="0.7514" style="stop-color:#CD2335"/>
+ <stop offset="1" style="stop-color:#E97826"/>
+</linearGradient>
+<path fill="url(#SVGID_10_)" d="M980.2,1186.7C980.2,1186.7,980.2,1186.7,980.2,1186.7L980.2,1186.7L980.2,1186.7L980.2,1186.7
+ C980.2,1186.7,980.2,1186.7,980.2,1186.7z"/>
+</svg>
diff --git a/content/images/hoodie_cow.png b/content/images/hoodie_cow.png
deleted file mode 100644
index bad15a8..0000000
Binary files a/content/images/hoodie_cow.png and /dev/null differ
diff --git a/content/images/hoodie_intro_1.png b/content/images/hoodie_intro_1.png
index dd8977a..6220b46 100644
Binary files a/content/images/hoodie_intro_1.png and b/content/images/hoodie_intro_1.png differ
diff --git a/content/images/hoodie_mor.png b/content/images/hoodie_mor.png
deleted file mode 100644
index 8d7d902..0000000
Binary files a/content/images/hoodie_mor.png and /dev/null differ
diff --git a/content/images/hudi_cow.png b/content/images/hudi_cow.png
new file mode 100644
index 0000000..40aca71
Binary files /dev/null and b/content/images/hudi_cow.png differ
diff --git a/content/images/hudi_mor.png b/content/images/hudi_mor.png
new file mode 100644
index 0000000..100b8f0
Binary files /dev/null and b/content/images/hudi_mor.png differ
diff --git a/content/images/hoodie_timeline.png b/content/images/hudi_timeline.png
similarity index 100%
rename from content/images/hoodie_timeline.png
rename to content/images/hudi_timeline.png
diff --git a/content/implementation.html b/content/implementation.html
index 1e9c4d6..d649a70 100644
--- a/content/implementation.html
+++ b/content/implementation.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -362,7 +363,7 @@ Hudi upsert/insert is merely a Spark DAG, that can be broken into two big pieces
<p>Hudi currently provides two choices for indexes : <code class="highlighter-rouge">BloomIndex</code> and <code class="highlighter-rouge">HBaseIndex</code> to map a record key into the file id to which it belongs to. This enables
us to speed up upserts significantly, without scanning over every record in the dataset. Hudi Indices can be classified based on
-their ability to lookup records across partition. A <code class="highlighter-rouge">global</code> index does not need partition information for finding the file-id for a record key
+their ability to lookup records across partition. A <code class="highlighter-rouge">global</code> index does not need partition information for finding the file-id for a record key
but a <code class="highlighter-rouge">non-global</code> does.</p>
<h4 id="hbase-index-global">HBase Index (global)</h4>
@@ -402,7 +403,7 @@ Any remaining records after that, are again packed into new file id groups, agai
<p>In this storage, index updation is a no-op, since the bloom filters are already written as a part of committing data.</p>
-<p>In the case of Copy-On-Write, a single parquet file constitutes one <code class="highlighter-rouge">file slice</code> which contains one complete version of
+<p>In the case of Copy-On-Write, a single parquet file constitutes one <code class="highlighter-rouge">file slice</code> which contains one complete version of
the file</p>
<figure><img class="docimage" src="images/hoodie_log_format_v2.png" alt="hoodie_log_format_v2.png" style="max-width: 1000px" /></figure>
@@ -416,7 +417,7 @@ the file</p>
<h5 id="merge-on-read-writer">Merge On Read Writer</h5>
<p>At a high level, Merge-On-Read Writer goes through same stages as Copy-On-Write writer in ingesting data.
- The key difference here is that updates are appended to latest log (delta) file belonging to the latest file slice
+ The key difference here is that updates are appended to latest log (delta) file belonging to the latest file slice
without merging. For inserts, Hudi supports 2 modes:</p>
<ol>
@@ -425,7 +426,7 @@ the file</p>
embedded in parquer files. Hudi treats writing new records in the same way as inserting to Copy-On-Write files.</li>
</ol>
-<p>As in the case of Copy-On-Write, the input tagged records are partitioned such that all upserts destined to
+<p>As in the case of Copy-On-Write, the input tagged records are partitioned such that all upserts destined to
a <code class="highlighter-rouge">file id</code> are grouped together. This upsert-batch is written as one or more log-blocks written to log-files.
Hudi allows clients to control log file sizes (See <a href="../configurations">Storage Configs</a>)</p>
@@ -448,54 +449,55 @@ compacting these log-files asynchronously to create read-optimized views.</p>
to be compacted atomically in a single compaction commit. Hudi allows pluggable strategies for choosing
file slices for each compaction runs. This step is typically done inline by Writer process as Hudi expects
only one schedule is being generated at a time which allows Hudi to enforce the constraint that pending compaction
-plans do not step on each other file-slices. This constraint allows for multiple concurrent <code class="highlighter-rouge">Compactors</code> to run at
+plans do not step on each other file-slices. This constraint allows for multiple concurrent <code class="highlighter-rouge">Compactors</code> to run at
the same time. Some of the common strategies used for choosing <code class="highlighter-rouge">file slice</code> for compaction are:
<ul>
- <li>BoundedIO - Limit the number of file slices chosen for a compaction plan by expected total IO (read + write)
+ <li>BoundedIO - Limit the number of file slices chosen for a compaction plan by expected total IO (read + write)
needed to complete compaction run</li>
<li>Log File Size - Prefer file-slices with larger amounts of delta log data to be merged</li>
- <li>Day Based - Prefer file slice belonging to latest day partitions
-<code class="highlighter-rouge">
- API for scheduling compaction
- /**
- * Schedules a new compaction instant
- * @param extraMetadata
- * @return Compaction Instant timestamp if a new compaction plan is scheduled
- */
- Optional<String> scheduleCompaction(Optional<Map<String, String>> extraMetadata) throws IOException;
-</code></li>
+ <li>Day Based - Prefer file slice belonging to latest day partitions</li>
</ul>
</li>
- <li>
- <p><code class="highlighter-rouge">Compactor</code> : Hudi provides a separate API in Write Client to execute a compaction plan. The compaction
+ <li><code class="highlighter-rouge">Compactor</code> : Hudi provides a separate API in Write Client to execute a compaction plan. The compaction
plan (just like a commit) is identified by a timestamp. Most of the design and implementation complexities for Async
Compaction is for guaranteeing snapshot isolation to readers and writer when
multiple concurrent compactors are running. Typical compactor deployment involves launching a separate
spark application which executes pending compactions when they become available. The core logic of compacting
file slices in the Compactor is very similar to that of merging updates in a Copy-On-Write table. The only
-difference being in the case of compaction, there is an additional step of merging the records in delta log-files.</p>
+difference being in the case of compaction, there is an additional step of merging the records in delta log-files.</li>
+</ul>
- <p>Here are the main API to lookup and execute a compaction plan.
-```
- Main API in HoodieWriteClient for running Compaction:
+<p>Here are the main API to lookup and execute a compaction plan.</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code> Main API in HoodieWriteClient for running Compaction:
/**
* Performs Compaction corresponding to instant-time
* @param compactionInstantTime Compaction Instant Time
* @return
* @throws IOException
*/
- public JavaRDD<WriteStatus> compact(String compactionInstantTime) throws IOException;</WriteStatus></p>
+ public JavaRDD<WriteStatus> compact(String compactionInstantTime) throws IOException;
- <p>To lookup all pending compactions, use the API defined in HoodieReadClient</p>
+ To lookup all pending compactions, use the API defined in HoodieReadClient
- <p>/**
+ /**
* Return all pending compactions with instant time for clients to decide what to compact next.
* @return
*/
- public List<Pair<String, HoodieCompactionPlan» getPendingCompactions();
-```</p>
- </li>
-</ul>
+ public List<Pair<String, HoodieCompactionPlan>> getPendingCompactions();
+</code></pre>
+</div>
+<p>API for scheduling compaction</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>
+ /**
+ * Schedules a new compaction instant
+ * @param extraMetadata
+ * @return Compaction Instant timestamp if a new compaction plan is scheduled
+ */
+ Optional<String> scheduleCompaction(Optional<Map<String, String>> extraMetadata) throws IOException;
+</code></pre>
+</div>
<p>Refer to <strong>hoodie-client/src/test/java/HoodieClientExample.java</strong> class for an example of how compaction
is scheduled and executed.</p>
@@ -525,61 +527,61 @@ plan to be run to figure out the number of file slices being compacted and choos
<p>For the purpose of this section, it is important to distinguish between 2 types of commits as pertaining to the file-group:</p>
<p>A commit which generates a merged and read-optimized file-slice is called <code class="highlighter-rouge">snapshot commit</code> (SC) with respect to that file-group.
-A commit which merely appended the new/updated records assigned to the file-group into a new log block is called <code class="highlighter-rouge">delta commit</code> (DC)
+A commit which merely appended the new/updated records assigned to the file-group into a new log block is called <code class="highlighter-rouge">delta commit</code> (DC)
with respect to that file-group.</p>
<h3 id="algorithm">Algorithm</h3>
<p>The algorithm is described with an illustration. Let us assume a scenario where there are commits SC1, DC2, DC3 that have
-already completed on a data-set. Commit DC4 is currently ongoing with the writer (ingestion) process using it to upsert data.
-Let us also imagine there are a set of file-groups (FG1 … FGn) in the data-set whose latest version (<code class="highlighter-rouge">File-Slice</code>)
-contains the base file created by commit SC1 (snapshot-commit in columnar format) and a log file containing row-based
+already completed on a data-set. Commit DC4 is currently ongoing with the writer (ingestion) process using it to upsert data.
+Let us also imagine there are a set of file-groups (FG1 … FGn) in the data-set whose latest version (<code class="highlighter-rouge">File-Slice</code>)
+contains the base file created by commit SC1 (snapshot-commit in columnar format) and a log file containing row-based
log blocks of 2 delta-commits (DC2 and DC3).</p>
<figure><img class="docimage" src="images/async_compac_1.png" alt="async_compac_1.png" style="max-width: 1000px" /></figure>
<ul>
- <li>Writer (Ingestion) that is going to commit “DC4” starts. The record updates in this batch are grouped by file-groups
-and appended in row formats to the corresponding log file as delta commit. Let us imagine a subset of file-groups has
+ <li>Writer (Ingestion) that is going to commit “DC4” starts. The record updates in this batch are grouped by file-groups
+and appended in row formats to the corresponding log file as delta commit. Let us imagine a subset of file-groups has
this new log block (delta commit) DC4 added.</li>
<li>
- <p>Before the writer job completes, it runs the compaction strategy to decide which file-group to compact by compactor
-and creates a new compaction-request commit SC5. This commit file is marked as “requested” with metadata denoting
+ <p>Before the writer job completes, it runs the compaction strategy to decide which file-group to compact by compactor
+and creates a new compaction-request commit SC5. This commit file is marked as “requested” with metadata denoting
which fileIds to compact (based on selection policy). Writer completes without running compaction (will be run async).</p>
<figure><img class="docimage" src="images/async_compac_2.png" alt="async_compac_2.png" style="max-width: 1000px" /></figure>
</li>
<li>
- <p>Writer job runs again ingesting next batch. It starts with commit DC6. It reads the earliest inflight compaction
-request marker commit in timeline order and collects the (fileId, Compaction Commit Id “CcId” ) pairs from meta-data.
-Ingestion DC6 ensures a new file-slice with base-commit “CcId” gets allocated for the file-group.
-The Writer will simply append records in row-format to the first log-file (as delta-commit) assuming the
+ <p>Writer job runs again ingesting next batch. It starts with commit DC6. It reads the earliest inflight compaction
+request marker commit in timeline order and collects the (fileId, Compaction Commit Id “CcId” ) pairs from meta-data.
+Ingestion DC6 ensures a new file-slice with base-commit “CcId” gets allocated for the file-group.
+The Writer will simply append records in row-format to the first log-file (as delta-commit) assuming the
base-file (“Phantom-Base-File”) will be created eventually by the compactor.</p>
<figure><img class="docimage" src="images/async_compac_3.png" alt="async_compac_3.png" style="max-width: 1000px" /></figure>
</li>
<li>
- <p>Compactor runs at some time and commits at “Tc” (concurrently or before/after Ingestion DC6). It reads the commit-timeline
-and finds the first unprocessed compaction request marker commit. Compactor reads the commit’s metadata finding the
-file-slices to be compacted. It compacts the file-slice and creates the missing base-file (“Phantom-Base-File”)
-with “CCId” as the commit-timestamp. Compactor then marks the compaction commit timestamp as completed.
-It is important to realize that at data-set level, there could be different file-groups requesting compaction at
+ <p>Compactor runs at some time and commits at “Tc” (concurrently or before/after Ingestion DC6). It reads the commit-timeline
+and finds the first unprocessed compaction request marker commit. Compactor reads the commit’s metadata finding the
+file-slices to be compacted. It compacts the file-slice and creates the missing base-file (“Phantom-Base-File”)
+with “CCId” as the commit-timestamp. Compactor then marks the compaction commit timestamp as completed.
+It is important to realize that at data-set level, there could be different file-groups requesting compaction at
different commit timestamps.</p>
<figure><img class="docimage" src="images/async_compac_4.png" alt="async_compac_4.png" style="max-width: 1000px" /></figure>
</li>
- <li>Near Real-time reader interested in getting the latest snapshot will have 2 cases. Let us assume that the
+ <li>Near Real-time reader interested in getting the latest snapshot will have 2 cases. Let us assume that the
incremental ingestion (writer at DC6) happened before the compaction (some time “Tc”’).<br />
The below description is with regards to compaction from file-group perspective.
<ul>
- <li><code class="highlighter-rouge">Reader querying at time between ingestion completion time for DC6 and compaction finish “Tc”</code>:
-Hoodie’s implementation will be changed to become aware of file-groups currently waiting for compaction and
-merge log-files corresponding to DC2-DC6 with the base-file corresponding to SC1. In essence, Hudi will create
-a pseudo file-slice by combining the 2 file-slices starting at base-commits SC1 and SC5 to one.
-For file-groups not waiting for compaction, the reader behavior is essentially the same - read latest file-slice
+ <li><code class="highlighter-rouge">Reader querying at time between ingestion completion time for DC6 and compaction finish “Tc”</code>:
+Hoodie’s implementation will be changed to become aware of file-groups currently waiting for compaction and
+merge log-files corresponding to DC2-DC6 with the base-file corresponding to SC1. In essence, Hudi will create
+a pseudo file-slice by combining the 2 file-slices starting at base-commits SC1 and SC5 to one.
+For file-groups not waiting for compaction, the reader behavior is essentially the same - read latest file-slice
and merge on the fly.</li>
- <li><code class="highlighter-rouge">Reader querying at time after compaction finished (> “Tc”)</code> : In this case, reader will not find any pending
-compactions in the timeline and will simply have the current behavior of reading the latest file-slice and
+ <li><code class="highlighter-rouge">Reader querying at time after compaction finished (> “Tc”)</code> : In this case, reader will not find any pending
+compactions in the timeline and will simply have the current behavior of reading the latest file-slice and
merging on-the-fly.</li>
</ul>
</li>
@@ -587,7 +589,7 @@ merging on-the-fly.</li>
</ul>
<p>The above algorithm explains Async compaction w.r.t a single compaction run on a single file-group. It is important
-to note that multiple compaction plans can be run concurrently as they are essentially operating on different
+to note that multiple compaction plans can be run concurrently as they are essentially operating on different
file-groups.</p>
<h2 id="performance">Performance</h2>
@@ -641,9 +643,19 @@ with no impact on queries. Following charts compare the Hudi vs non-Hudi dataset
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/incremental_processing.html b/content/incremental_processing.html
index f4d31da..a694881 100644
--- a/content/incremental_processing.html
+++ b/content/incremental_processing.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -421,7 +422,12 @@ Usage: <main class> [options]
exist first time around. If exists, expected to be a Hudi dataset)
* --target-table
name of the target table in Hive
-
+ --transformer-class
+ subclass of com.uber.hoodie.utilities.transform.Transformer. UDF to
+ transform raw source dataset to a target dataset (conforming to target
+ schema) before writing. Default : Not set. E:g -
+ com.uber.hoodie.utilities.transform.SqlQueryBasedTransformer (which
+ allows a SQL query template to be passed as a transformation function)
</code></pre>
</div>
@@ -655,9 +661,19 @@ is the lack of support for self-joining the same table in mixed mode (normal and
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/index.html b/content/index.html
index 4ffe717..bd31b4d 100644
--- a/content/index.html
+++ b/content/index.html
@@ -3,9 +3,9 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
-<meta name="description" content="Hudi lowers data latency across the board, while simultaneously achieving orders of magnitude of efficiency over traditional batch processing.">
+<meta name="description" content="Hudi brings stream processing to big data, providing fresh data while being an order of magnitude efficient over traditional batch processing.">
<meta name="keywords" content="getting_started, homepage">
-<title>Apache Hudi Overview | Hudi</title>
+<title>What is Hudi? | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -157,7 +158,7 @@
- <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Apache Hudi Overview page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
+ <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the What is Hudi? page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
<li>
@@ -175,7 +176,7 @@
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}" title="Apache Hudi Overview">{title}</a></li>',
+ searchResultTemplate: '<li><a href="{url}" title="What is Hudi?">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
@@ -326,7 +327,7 @@
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
- <h1 class="post-title-main">Apache Hudi Overview</h1>
+ <h1 class="post-title-main">What is Hudi?</h1>
</div>
@@ -334,7 +335,7 @@
<div class="post-content">
- <div class="summary">Hudi lowers data latency across the board, while simultaneously achieving orders of magnitude of efficiency over traditional batch processing.</div>
+ <div class="summary">Hudi brings stream processing to big data, providing fresh data while being an order of magnitude efficient over traditional batch processing.</div>
@@ -365,21 +366,19 @@ $('#toc').on('click', 'a', function() {
- <p>Hudi (pronounced “Hoodie”) manages storage of large analytical datasets on <a href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">HDFS</a> or cloud and serves them out via two types of tables</p>
+ <p>Hudi (pronounced “Hoodie”) ingests & manages storage of large analytical datasets on <a href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">HDFS</a> or cloud stores and provides three logical views for query access.</p>
<ul>
- <li><strong>Read Optimized Table</strong> - Provides excellent query performance via purely columnar storage (e.g. <a href="https://parquet.apache.org/">Parquet</a>)</li>
+ <li><strong>Read Optimized View</strong> - Provides excellent query performance on pure columnar storage, much like plain <a href="https://parquet.apache.org/">Parquet</a> tables.</li>
+ <li><strong>Incremental View</strong> - Provides a change stream out of the dataset to feed downstream jobs/ETLs.</li>
<li><strong>Near-Real time Table</strong> - Provides queries on real-time data, using a combination of columnar & row based storage (e.g Parquet + <a href="http://avro.apache.org/docs/current/mr.html">Avro</a>)</li>
</ul>
<figure><img class="docimage" src="images/hoodie_intro_1.png" alt="hoodie_intro_1.png" /></figure>
-<p>By carefully managing how data is laid out in storage & how it’s exposed to queries, Hoodie is able to power a rich data ecosystem where external sources can be ingested into Hadoop in near real-time. The ingested data is then available for interactive SQL Engines like <a href="https://prestodb.io">Presto</a> & <a href="https://spark.apache.org/sql/">Spark</a>, while at the same time capable of being consumed incrementally from processing/ETL frameworks like <a href="https://h [...]
-
-<p>Hudi broadly consists of a self contained Spark library to build datasets and integrations with existing query engines for data access.</p>
-
-<div class="bs-callout bs-callout-info">Hudi is a new project. Near Real-Time Table implementation is currently underway. Get involved <a href="https://github.com/uber/hoodie/projects/1">here</a></div>
+<p>By carefully managing how data is laid out in storage & how it’s exposed to queries, Hudi is able to power a rich data ecosystem where external sources can be ingested in near real-time and made available for interactive SQL Engines like <a href="https://prestodb.io">Presto</a> & <a href="https://spark.apache.org/sql/">Spark</a>, while at the same time capable of being consumed incrementally from processing/ETL frameworks like <a href="https://hive.apache.org/">Hive</a> & [...]
+<p>Hudi broadly consists of a self contained Spark library to build datasets and integrations with existing query engines for data access. See <a href="quickstart.html">quickstart</a> for a demo.</p>
<div class="tags">
@@ -403,9 +402,19 @@ $('#toc').on('click', 'a', function() {
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/js/mydoc_scroll.html b/content/js/mydoc_scroll.html
index bc22ec4..b23a6ad 100644
--- a/content/js/mydoc_scroll.html
+++ b/content/js/mydoc_scroll.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -615,9 +616,19 @@ $('#small-box-links').localScroll({
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
-<span>Page last updated:</span> November 30, 2015<br/> Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/migration_guide.html b/content/migration_guide.html
index 8b1793e..7bcfa1d 100644
--- a/content/migration_guide.html
+++ b/content/migration_guide.html
@@ -3,9 +3,9 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
-<meta name="description" content="">
-<meta name="keywords" content=" ">
-<title> | Hudi</title>
+<meta name="description" content="In this page, we will discuss some available tools for migrating your existing dataset into a Hudi dataset">
+<meta name="keywords" content=" migration guide">
+<title>Migration Guide | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -157,7 +158,7 @@
- <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
+ <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Migration Guide page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
<li>
@@ -175,7 +176,7 @@
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}" title="">{title}</a></li>',
+ searchResultTemplate: '<li><a href="{url}" title="Migration Guide">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
@@ -326,7 +327,7 @@
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
- <h1 class="post-title-main"></h1>
+ <h1 class="post-title-main">Migration Guide</h1>
</div>
@@ -334,99 +335,72 @@
<div class="post-content">
-
-
-
-<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
-<script>
-$( document ).ready(function() {
- // Handler for .ready() called.
-
-$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
-
-/* this offset helps account for the space taken up by the floating toolbar. */
-$('#toc').on('click', 'a', function() {
- var target = $(this.getAttribute('href'))
- , scroll_target = target.offset().top
-
- $(window).scrollTop(scroll_target - 10);
- return false
-})
-
-});
-</script>
-
-<div id="toc"></div>
+ <div class="summary">In this page, we will discuss some available tools for migrating your existing dataset into a Hudi dataset</div>
+
- <hr />
-<p>title: Migration Guide
-keywords: migration guide
-sidebar: mydoc_sidebar
-permalink: migration_guide.html
-toc: false
-summary: In this page, we will discuss some available tools for migrating your existing dataset into a Hudi managed
-dataset</p>
-
-<p>Hudi maintains metadata such as commit timeline and indexes to manage a dataset. The commit timelines helps to understand the actions happening on a dataset as well as the current state of a dataset. Indexes are used by Hudi to maintain a record key to file id mapping to efficiently locate a record. At the moment, Hudi supports writing only parquet columnar formats.
+ <p>Hudi maintains metadata such as commit timeline and indexes to manage a dataset. The commit timelines helps to understand the actions happening on a dataset as well as the current state of a dataset. Indexes are used by Hudi to maintain a record key to file id mapping to efficiently locate a record. At the moment, Hudi supports writing only parquet columnar formats.
To be able to start using Hudi for your existing dataset, you will need to migrate your existing dataset into a Hudi managed dataset. There are a couple of ways to achieve this.</p>
<h2 id="approaches">Approaches</h2>
-<h3 id="approach-1">Approach 1</h3>
+<h4 id="use-hudi-for-new-partitions-alone">Use Hudi for new partitions alone</h4>
-<p>Hudi can be used to manage an existing dataset without affecting/altering the historical data already present in the
-dataset. Hudi has been implemented to be compatible with such a mixed dataset with a caveat that either the complete
-Hive partition is Hudi managed or not. Thus the lowest granularity at which Hudi manages a dataset is a Hive
-partition. Start using the datasource API or the WriteClient to write to the dataset and make sure you start writing
+<p>Hudi can be used to manage an existing dataset without affecting/altering the historical data already present in the
+dataset. Hudi has been implemented to be compatible with such a mixed dataset with a caveat that either the complete
+Hive partition is Hudi managed or not. Thus the lowest granularity at which Hudi manages a dataset is a Hive
+partition. Start using the datasource API or the WriteClient to write to the dataset and make sure you start writing
to a new partition or convert your last N partitions into Hudi instead of the entire table. Note, since the historical
- partitions are not managed by HUDI, none of the primitives provided by HUDI work on the data in those partitions. More concretely, one cannot perform upserts or incremental pull on such older partitions not managed by the HUDI dataset.
+ partitions are not managed by HUDI, none of the primitives provided by HUDI work on the data in those partitions. More concretely, one cannot perform upserts or incremental pull on such older partitions not managed by the HUDI dataset.
Take this approach if your dataset is an append only type of dataset and you do not expect to perform any updates to existing (or non Hudi managed) partitions.</p>
-<h3 id="approach-2">Approach 2</h3>
+<h4 id="convert-existing-dataset-to-hudi">Convert existing dataset to Hudi</h4>
<p>Import your existing dataset into a Hudi managed dataset. Since all the data is Hudi managed, none of the limitations
- of Approach 1 apply here. Updates spanning any partitions can be applied to this dataset and Hudi will efficiently
- make the update available to queries. Note that not only do you get to use all Hoodie primitives on this dataset,
+ of Approach 1 apply here. Updates spanning any partitions can be applied to this dataset and Hudi will efficiently
+ make the update available to queries. Note that not only do you get to use all Hoodie primitives on this dataset,
there are other additional advantages of doing this. Hudi automatically manages file sizes of a Hudi managed dataset
- . You can define the desired file size when converting this dataset and Hudi will ensure it writes out files
- adhering to the config. It will also ensure that smaller files later get corrected by routing some new inserts into
+ . You can define the desired file size when converting this dataset and Hudi will ensure it writes out files
+ adhering to the config. It will also ensure that smaller files later get corrected by routing some new inserts into
small files rather than writing new small ones thus maintaining the health of your cluster.</p>
-<p>There are a few options when choosing this approach.
-#### Option 1
-Use the HDFSParquetImporter tool. As the name suggests, this only works if your existing dataset is in
-parquet file
-format. This tool essentially starts a Spark Job to read the existing parquet dataset and converts it into a HUDI managed dataset by re-writing all the data.
-#### Option 2
-For huge datasets, this could be as simple as : for partition in [list of partitions in source dataset] {
+<p>There are a few options when choosing this approach.</p>
+
+<h4 id="option-1">Option 1</h4>
+<p>Use the HDFSParquetImporter tool. As the name suggests, this only works if your existing dataset is in
+parquet file
+format. This tool essentially starts a Spark Job to read the existing parquet dataset and converts it into a HUDI managed dataset by re-writing all the data.</p>
+
+<h4 id="option-2">Option 2</h4>
+<p>For huge datasets, this could be as simple as : for partition in [list of partitions in source dataset] {
val inputDF = spark.read.format(“any_input_format”).load(“partition_path”)
inputDF.write.format(“com.uber.hoodie”).option()….save(“basePath”)
- } <br />
-#### Option 3
-Write your own custom logic of how to load an existing dataset into a Hudi managed one. Please read about the RDD API
- <a href="/quickstart.html">here</a>.</p>
+ }</p>
+
+<h4 id="option-3">Option 3</h4>
+<p>Write your own custom logic of how to load an existing dataset into a Hudi managed one. Please read about the RDD API
+ <a href="quickstart.html">here</a>.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>Using the HDFSParquetImporter Tool. Once hoodie has been built via `mvn clean install -DskipTests`, the shell can be
+<div class="highlighter-rouge"><pre class="highlight"><code>Using the HDFSParquetImporter Tool. Once hoodie has been built via `mvn clean install -DskipTests`, the shell can be
fired by via `cd hoodie-cli && ./hoodie-cli.sh`.
-hoodie->hdfsparquetimport
- --upsert false
- --srcPath /user/parquet/dataset/basepath
- --targetPath
- /user/hoodie/dataset/basepath
- --tableName hoodie_table
- --tableType COPY_ON_WRITE
- --rowKeyField _row_key
- --partitionPathField partitionStr
- --parallelism 1500
- --schemaFilePath /user/table/schema
- --format parquet
- --sparkMemory 6g
+hoodie->hdfsparquetimport
+ --upsert false
+ --srcPath /user/parquet/dataset/basepath
+ --targetPath
+ /user/hoodie/dataset/basepath
+ --tableName hoodie_table
+ --tableType COPY_ON_WRITE
+ --rowKeyField _row_key
+ --partitionPathField partitionStr
+ --parallelism 1500
+ --schemaFilePath /user/table/schema
+ --format parquet
+ --sparkMemory 6g
--retry 2
</code></pre>
</div>
@@ -445,9 +419,19 @@ hoodie->hdfsparquetimport
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/news.html b/content/news.html
index b124601..645bae0 100644
--- a/content/news.html
+++ b/content/news.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -244,6 +245,20 @@
+ <h2><a class="post-link" href="strata-talk.html">Hudi entered Apache Incubator</a></h2>
+ <span class="post-meta">Jan 18, 2019 /
+
+
+ <a href="tag_news.html">news</a>
+
+ </span>
+ <p> In the coming weeks, we will be moving in our new home on the Apache Incubator.
+
+ </p>
+
+
+
+
<h2><a class="post-link" href="strata-talk.html">Connect with us at Strata San Jose March 2017</a></h2>
<span class="post-meta">Dec 30, 2016 /
@@ -280,9 +295,19 @@ Catch our talk “Incremental Processing on Hadoop At Uber”
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/news_archive.html b/content/news_archive.html
index fd98f67..4d80715 100644
--- a/content/news_archive.html
+++ b/content/news_archive.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -351,6 +352,17 @@
<ul class="this">
+ <li><time>18 Jan</time><a href="strata-talk.html">Hudi entered Apache Incubator</a></li>
+
+
+
+
+
+ </ul>
+ <h3>2016</h3>
+ <ul class="past">
+
+
<li><time>30 Dec</time><a href="strata-talk.html">Connect with us at Strata San Jose March 2017</a></li>
</ul>
@@ -379,9 +391,19 @@
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/powered_by.html b/content/powered_by.html
index 2bced47..8f4b0d4 100644
--- a/content/powered_by.html
+++ b/content/powered_by.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -396,9 +397,19 @@ October 2018, Spark+AI Summit Europe, London, UK</p>
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/code_and_design.html b/content/privacy.html
similarity index 74%
rename from content/code_and_design.html
rename to content/privacy.html
index 90372b2..704bd3d 100644
--- a/content/code_and_design.html
+++ b/content/privacy.html
@@ -4,8 +4,8 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
-<meta name="keywords" content=" usecases">
-<title>Code Structure | Hudi</title>
+<meta name="keywords" content=" privacy">
+<title>Privacy Policy | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li class="dropdownActive"><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -157,7 +158,7 @@
- <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Code Structure page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
+ <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Privacy Policy page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
<li>
@@ -175,7 +176,7 @@
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}" title="Code Structure">{title}</a></li>',
+ searchResultTemplate: '<li><a href="{url}" title="Privacy Policy">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
@@ -326,7 +327,7 @@
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
- <h1 class="post-title-main">Code Structure</h1>
+ <h1 class="post-title-main">Privacy Policy</h1>
</div>
@@ -363,71 +364,24 @@ $('#toc').on('click', 'a', function() {
- <h2 id="code--project-structure">Code & Project Structure</h2>
+ <p>Information about your use of this website is collected using server access logs and a tracking cookie.
+The collected information consists of the following:</p>
<ul>
- <li>hoodie-client : Spark client library to take a bunch of inserts + updates and apply them to a Hoodie table</li>
- <li>hoodie-common : Common code shared between different artifacts of Hoodie</li>
+ <li>The IP address from which you access the website;</li>
+ <li>The type of browser and operating system you use to access our site;</li>
+ <li>The date and time you access our site;</li>
+ <li>The pages you visit;</li>
+ <li>The addresses of pages from where you followed a link to our site.</li>
</ul>
-<p>## HoodieLogFormat</p>
-
-<p>The following diagram depicts the LogFormat for Hoodie MergeOnRead. Each logfile consists of one or more log blocks.
- Each logblock follows the format shown below.</p>
-
-<table>
- <thead>
- <tr>
- <th>Field</th>
- <th>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>MAGIC</td>
- <td>A magic header that marks the start of a block</td>
- </tr>
- <tr>
- <td>VERSION</td>
- <td>The version of the LogFormat, this helps define how to switch between different log format as it evolves</td>
- </tr>
- <tr>
- <td>TYPE</td>
- <td>The type of the log block</td>
- </tr>
- <tr>
- <td>HEADER LENGTH</td>
- <td>The length of the headers, 0 if no headers</td>
- </tr>
- <tr>
- <td>HEADER</td>
- <td>Metadata needed for a log block. For eg. INSTANT_TIME, TARGET_INSTANT_TIME, SCHEMA etc.</td>
- </tr>
- <tr>
- <td>CONTENT LENGTH</td>
- <td>The length of the content of the log block</td>
- </tr>
- <tr>
- <td>CONTENT</td>
- <td>The content of the log block, for example, for a DATA_BLOCK, the content is (number of records + actual records) in byte []</td>
- </tr>
- <tr>
- <td>FOOTER LENGTH</td>
- <td>The length of the footers, 0 if no footers</td>
- </tr>
- <tr>
- <td>FOOTER</td>
- <td>Metadata needed for a log block. For eg. index entries, a bloom filter for records in a DATA_BLOCK etc.</td>
- </tr>
- <tr>
- <td>LOGBLOCK LENGTH</td>
- <td>The total number of bytes written for a log block, typically the SUM(everything_above). This is a LONG. This acts as a reverse pointer to be able to traverse the log in reverse.</td>
- </tr>
- </tbody>
-</table>
-
-<figure><img class="docimage" src="images/hoodie_log_format_v2.png" alt="hoodie_log_format_v2.png" /></figure>
+<p>Part of this information is gathered using a tracking cookie set by the <a href="http://www.google.com/analytics">Google Analytics</a> service and handled by Google as described in their <a href="http://www.google.com/privacy.html">privacy policy</a>. See your browser documentation for instructions on how to disable the cookie if you prefer not to share this data with Google.</p>
+<p>We use the gathered information to help us make our site more useful to visitors and to better understand how and when our site is used. We do not track or collect personally identifiable information or associate gathered data with any personally identifying information from other sources.</p>
+
+<p>By using this website, you consent to the collection of this data in the manner and for the purpose described above.</p>
+
+<p>The Hudi development community welcomes your questions or comments regarding this Privacy Policy. Send them to dev@hudi.apache.org</p>
<div class="tags">
@@ -443,9 +397,19 @@ $('#toc').on('click', 'a', function() {
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/quickstart.html b/content/quickstart.html
index fcd4524..a73534d 100644
--- a/content/quickstart.html
+++ b/content/quickstart.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -344,44 +345,66 @@
<p>Check out code and pull it into Intellij as a normal maven project.</p>
-<p>Normally build the maven project, from command line
-```
-$ mvn clean install -DskipTests -DskipITs</p>
-
-<p>To work with older version of Hive (pre Hive-1.2.1), use</p>
-
-<p>$ mvn clean install -DskipTests -DskipITs -Dhive11</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>
-<div markdown="span" class="bs-callout bs-callout-info">You might want to add your spark jars folder to project dependencies under 'Module Setttings', to be able to run Spark from IDE</div>
-
-
-<div markdown="span" class="alert alert-info" role="alert"><i class="fa fa-info-circle"></i> <b>Note:</b> Setup your local hadoop/hive test environment, so you can play with entire ecosystem. See [this](http://www.bytearray.io/2016/05/setting-up-hadoopyarnsparkhive-on-mac.html) for reference</div>
-
-
-
-## Version Compatibility
-
-Hudi requires Java 8 to be installed. Hudi works with Spark-2.x versions. We have verified that Hudi works with the following combination of Hadoop/Hive/Spark.
-
-| Hadoop | Hive | Spark | Instructions to Build Hudi |
-| ---- | ----- | ---- | ---- |
-| 2.6.0-cdh5.7.2 | 1.1.0-cdh5.7.2 | spark-2.[1-3].x | Use “mvn clean install -DskipTests -Dhadoop.version=2.6.0-cdh5.7.2 -Dhive.version=1.1.0-cdh5.7.2” |
-| Apache hadoop-2.8.4 | Apache hive-2.3.3 | spark-2.[1-3].x | Use "mvn clean install -DskipTests" |
-| Apache hadoop-2.7.3 | Apache hive-1.2.1 | spark-2.[1-3].x | Use "mvn clean install -DskipTests" |
-
-If your environment has other versions of hadoop/hive/spark, please try out Hudi and let us know if there are any issues. We are limited by our bandwidth to certify other combinations.
-It would be of great help if you can reach out to us with your setup and experience with hoodie.
-
-## Generate a Hudi Dataset
-
-### Requirements & Environment Variable
-
-Please set the following environment variablies according to your setup. We have given an example setup with CDH version
+<p>Normally build the maven project, from command line</p>
+<div class="highlighter-rouge"><pre class="highlight"><code>$ mvn clean install -DskipTests -DskipITs
</code></pre>
</div>
-<p>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
+
+<p>To work with older version of Hive (pre Hive-1.2.1), use
+<code class="highlighter-rouge">
+$ mvn clean install -DskipTests -DskipITs -Dhive11
+</code></p>
+
+<div class="bs-callout bs-callout-info">You might want to add your spark jars folder to project dependencies under ‘Module Setttings’, to be able to run Spark from IDE</div>
+
+<div class="alert alert-info" role="alert"><i class="fa fa-info-circle"></i> <b>Note:</b> Setup your local hadoop/hive test environment, so you can play with entire ecosystem. See <a href="http://www.bytearray.io/2016/05/setting-up-hadoopyarnsparkhive-on-mac.html">this</a> for reference</div>
+
+<h2 id="version-compatibility">Version Compatibility</h2>
+
+<p>Hudi requires Java 8 to be installed. Hudi works with Spark-2.x versions. We have verified that Hudi works with the following combination of Hadoop/Hive/Spark.</p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Hadoop</th>
+ <th>Hive</th>
+ <th>Spark</th>
+ <th>Instructions to Build Hudi</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>2.6.0-cdh5.7.2</td>
+ <td>1.1.0-cdh5.7.2</td>
+ <td>spark-2.[1-3].x</td>
+ <td>Use “mvn clean install -DskipTests -Dhadoop.version=2.6.0-cdh5.7.2 -Dhive.version=1.1.0-cdh5.7.2”</td>
+ </tr>
+ <tr>
+ <td>Apache hadoop-2.8.4</td>
+ <td>Apache hive-2.3.3</td>
+ <td>spark-2.[1-3].x</td>
+ <td>Use “mvn clean install -DskipTests”</td>
+ </tr>
+ <tr>
+ <td>Apache hadoop-2.7.3</td>
+ <td>Apache hive-1.2.1</td>
+ <td>spark-2.[1-3].x</td>
+ <td>Use “mvn clean install -DskipTests”</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>If your environment has other versions of hadoop/hive/spark, please try out Hudi and let us know if there are any issues. We are limited by our bandwidth to certify other combinations.
+It would be of great help if you can reach out to us with your setup and experience with hoodie.</p>
+
+<h2 id="generate-a-hudi-dataset">Generate a Hudi Dataset</h2>
+
+<h3 id="requirements--environment-variable">Requirements & Environment Variable</h3>
+
+<p>Please set the following environment variablies according to your setup. We have given an example setup with CDH version</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
export HIVE_HOME=/var/hadoop/setup/apache-hive-1.1.0-cdh5.7.2-bin
export HADOOP_HOME=/var/hadoop/setup/hadoop-2.6.0-cdh5.7.2
export HADOOP_INSTALL=/var/hadoop/setup/hadoop-2.6.0-cdh5.7.2
@@ -390,11 +413,12 @@ export SPARK_HOME=/var/hadoop/setup/spark-2.3.1-bin-hadoop2.7
export SPARK_INSTALL=$SPARK_HOME
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PATH=$JAVA_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$SPARK_INSTALL/bin:$PATH
-```</p>
+</code></pre>
+</div>
<h3 id="supported-apis">Supported API’s</h3>
-<p>Use the DataSource API to quickly start reading or writing Hudi datasets in few lines of code. Ideal for most
+<p>Use the DataSource API to quickly start reading or writing Hudi datasets in few lines of code. Ideal for most
ingestion use-cases.
Use the RDD API to perform more involved actions on a Hudi dataset</p>
@@ -462,11 +486,11 @@ This can be run as frequently as the ingestion pipeline to make sure new partiti
<div class="highlighter-rouge"><pre class="highlight"><code>cd hoodie-hive
./run_sync_tool.sh
--user hive
- --pass hive
- --database default
- --jdbc-url "jdbc:hive2://localhost:10010/"
- --base-path tmp/hoodie/sample-table/
- --table hoodie_test
+ --pass hive
+ --database default
+ --jdbc-url "jdbc:hive2://localhost:10010/"
+ --base-path tmp/hoodie/sample-table/
+ --table hoodie_test
--partitioned-by field1,field2
</code></pre>
@@ -713,8 +737,8 @@ varadarb-C02SG7Q3G8WP:docker varadarb$ docker ps
<p>Stock Tracker data will be used to showcase both different Hudi Views and the effects of Compaction.</p>
-<p>Take a look at the directory <code class="highlighter-rouge">docker/demo/data</code>. There are 2 batches of stock data - each at 1 minute granularity.
-The first batch contains stocker tracker data for some stock symbols during the first hour of trading window
+<p>Take a look at the directory <code class="highlighter-rouge">docker/demo/data</code>. There are 2 batches of stock data - each at 1 minute granularity.
+The first batch contains stocker tracker data for some stock symbols during the first hour of trading window
(9:30 a.m to 10:30 a.m). The second batch contains tracker data for next 30 mins (10:30 - 11 a.m). Hudi will
be used to ingest these batches to a dataset which will contain the latest stock tracker data at hour level granularity.
The batches are windowed intentionally so that the second batch contains updates to some of the rows in the first batch.</p>
@@ -723,7 +747,7 @@ The batches are windowed intentionally so that the second batch contains updates
<p>Upload the first batch to Kafka topic ‘stock ticks’</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P
+<div class="highlighter-rouge"><pre class="highlight"><code>cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P
To check if the new topic shows up, use
kafkacat -b kafkabroker -L -J | jq .
@@ -771,7 +795,7 @@ kafkacat -b kafkabroker -L -J | jq .
<p>Hudi comes with a tool named DeltaStreamer. This tool can connect to variety of data sources (including Kafka) to
pull changes and apply to Hudi dataset using upsert/insert primitives. Here, we will use the tool to download
-json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool
+json data from kafka topic and ingest to both COW and MOR tables we initialized in the previous step. This tool
automatically initializes the datasets in the file-system if they do not exist yet.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker exec -it adhoc-2 /bin/bash
@@ -796,7 +820,7 @@ exit
</code></pre>
</div>
-<p>You can use HDFS web-browser to look at the datasets
+<p>You can use HDFS web-browser to look at the datasets
<code class="highlighter-rouge">http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow</code>.</p>
<p>You can explore the new partition folder created in the dataset along with a “deltacommit”
@@ -839,7 +863,7 @@ provides the ReadOptimized view for the Hudi dataset and the later provides the
<h4 id="step-4-a-run-hive-queries">Step 4 (a): Run Hive Queries</h4>
<p>Run a hive query to find the latest timestamp ingested for stock symbol ‘GOOG’. You will notice that both read-optimized
-(for both COW and MOR dataset)and realtime views (for MOR dataset)give the same value “10:29 a.m” as Hudi create a
+(for both COW and MOR dataset)and realtime views (for MOR dataset)give the same value “10:29 a.m” as Hudi create a
parquet file for the first batch of data.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker exec -it adhoc-2 /bin/bash
@@ -892,7 +916,7 @@ Now, run a projection query:
# Merge-On-Read Queries:
==========================
-Lets run similar queries against M-O-R dataset. Lets look at both
+Lets run similar queries against M-O-R dataset. Lets look at both
ReadOptimized and Realtime views supported by M-O-R dataset
# Run against ReadOptimized View. Notice that the latest timestamp is 10:29
@@ -997,7 +1021,7 @@ scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, clo
# Merge-On-Read Queries:
==========================
-Lets run similar queries against M-O-R dataset. Lets look at both
+Lets run similar queries against M-O-R dataset. Lets look at both
ReadOptimized and Realtime views supported by M-O-R dataset
# Run against ReadOptimized View. Notice that the latest timestamp is 10:29
@@ -1044,7 +1068,7 @@ scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, clo
<p>Upload the second batch of data and ingest this batch using delta-streamer. As this batch does not bring in any new
partitions, there is no need to run hive-sync</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>cat docker/demo/data/batch_2.json | kafkacat -b kafkabroker -t stock_ticks -P
+<div class="highlighter-rouge"><pre class="highlight"><code>cat docker/demo/data/batch_2.json | kafkacat -b kafkabroker -t stock_ticks -P
# Within Docker container, run the ingestion command
docker exec -it adhoc-2 /bin/bash
@@ -1061,15 +1085,15 @@ exit
<p>With Copy-On-Write table, the second ingestion by DeltaStreamer resulted in a new version of Parquet file getting created.
See <code class="highlighter-rouge">http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_cow/2018/08/31</code></p>
-<p>With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
+<p>With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
Take a look at the HDFS filesystem to get an idea: <code class="highlighter-rouge">http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor/2018/08/31</code></p>
<h4 id="step-6a-run-hive-queries">Step 6(a): Run Hive Queries</h4>
-<p>With Copy-On-Write table, the read-optimized view immediately sees the changes as part of second batch once the batch
+<p>With Copy-On-Write table, the read-optimized view immediately sees the changes as part of second batch once the batch
got committed as each ingestion creates newer versions of parquet files.</p>
-<p>With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
+<p>With Merge-On-Read table, the second ingestion merely appended the batch to an unmerged delta (log) file.
This is the time, when ReadOptimized and Realtime views will provide different results. ReadOptimized view will still
return “10:29 am” as it will only read from the Parquet file. Realtime View will do on-the-fly merge and return
latest committed data which is “10:59 a.m”.</p>
@@ -1099,7 +1123,7 @@ WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the futu
As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
-# Merge On Read Table:
+# Merge On Read Table:
# Read Optimized View
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG';
@@ -1169,7 +1193,7 @@ scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, clo
As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
-# Merge On Read Table:
+# Merge On Read Table:
# Read Optimized View
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor group by symbol HAVING symbol = 'GOOG'").show(100, false)
@@ -1235,8 +1259,8 @@ To show the effects of incremental-query, let us assume that a reader has alread
ingesting first batch. Now, for the reader to see effect of the second batch, he/she has to keep the start timestamp to
the commit time of the first batch (20180924064621) and run incremental query
-`Hudi incremental mode` provides efficient scanning for incremental queries by filtering out files that do not have any
-candidate rows using hudi-managed metadata.
+`Hudi incremental mode` provides efficient scanning for incremental queries by filtering out files that do not have any
+candidate rows using hudi-managed metadata.
</code></pre>
</div>
@@ -1333,7 +1357,7 @@ hoodie:stock_ticks_mor->compactions show all
___________________________________________________________________
| Compaction Instant Time| State | Total FileIds to be Compacted|
|==================================================================|
-
+
# Schedule a compaction. This will use Spark Launcher to schedule compaction
hoodie:stock_ticks_mor->compaction schedule
....
@@ -1353,7 +1377,7 @@ hoodie:stock_ticks_mor->compactions show all
___________________________________________________________________
| Compaction Instant Time| State | Total FileIds to be Compacted|
|==================================================================|
- | 20180924070031 | REQUESTED| 1 |
+ | 20180924070031 | REQUESTED| 1 |
# Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query
hoodie:stock_ticks_mor->compaction run --compactionInstant 20180924070031 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1
@@ -1494,8 +1518,8 @@ scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, clo
<code class="highlighter-rouge">
$ mvn pre-integration-test -DskipTests
</code>
-The above command builds docker images for all the services with
-current Hudi source installed at /var/hoodie/ws and also brings up the services using a compose file. We
+The above command builds docker images for all the services with
+current Hudi source installed at /var/hoodie/ws and also brings up the services using a compose file. We
currently use Hadoop (v2.8.4), Hive (v2.3.3) and Spark (v2.3.1) in docker images.</p>
<p>To bring down the containers
@@ -1510,9 +1534,9 @@ $ cd hoodie-integ-test
$ mvn docker-compose:up -DdetachedMode=true
</code></p>
-<p>Hudi is a library that is operated in a broader data analytics/ingestion environment
+<p>Hudi is a library that is operated in a broader data analytics/ingestion environment
involving Hadoop, Hive and Spark. Interoperability with all these systems is a key objective for us. We are
-actively adding integration-tests under <strong>hoodie-integ-test/src/test/java</strong> that makes use of this
+actively adding integration-tests under <strong>hoodie-integ-test/src/test/java</strong> that makes use of this
docker environment (See <strong>hoodie-integ-test/src/test/java/com/uber/hoodie/integ/ITTestHoodieSanity.java</strong> )</p>
<h4 id="building-local-docker-containers">Building Local Docker Containers:</h4>
@@ -1528,8 +1552,8 @@ inbuilt jars by mounting local HUDI workspace over the docker location</li>
</ol>
<p>This helps avoid maintaining separate docker images and avoids the costly step of building HUDI docker images locally.
-But if users want to test hudi from locations with lower network bandwidth, they can still build local images
-run the script
+But if users want to test hudi from locations with lower network bandwidth, they can still build local images
+run the script
<code class="highlighter-rouge">docker/build_local_docker_images.sh</code> to build local docker images before running <code class="highlighter-rouge">docker/setup_demo.sh</code></p>
<p>Here are the commands:</p>
@@ -1594,9 +1618,19 @@ run the script
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/roadmap.html b/content/roadmap.html
deleted file mode 100644
index be1affe..0000000
--- a/content/roadmap.html
+++ /dev/null
@@ -1,405 +0,0 @@
-<!DOCTYPE html>
-<head>
- <meta charset="utf-8">
-<meta http-equiv="X-UA-Compatible" content="IE=edge">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<meta name="description" content="">
-<meta name="keywords" content=" usecases">
-<title>Roadmap | Hudi</title>
-<link rel="stylesheet" href="css/syntax.css">
-
-
-<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
-<!--<link rel="stylesheet" type="text/css" href="css/bootstrap.min.css">-->
-<link rel="stylesheet" href="css/modern-business.css">
-<link rel="stylesheet" href="css/lavish-bootstrap.css">
-<link rel="stylesheet" href="css/customstyles.css">
-<link rel="stylesheet" href="css/theme-blue.css">
-
-<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
-<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js"></script>
-<script src="js/jquery.navgoco.min.js"></script>
-
-
-<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
-<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js"></script>
-<script src="js/toc.js"></script>
-<script src="js/customscripts.js"></script>
-
-<script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
-
- ga('create', 'UA-93561550-1', 'auto');
- ga('send', 'pageview');
-
-</script>
-
-<link rel="shortcut icon" href="images/favicon.ico">
-
-<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
-<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
-<!--[if lt IE 9]>
-<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
-<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
-<![endif]-->
-
-<link rel="alternate" type="application/rss+xml" title="" href="http://0.0.0.0:4000feed.xml">
-
- <script>
- $(document).ready(function() {
- // Initialize navgoco with default options
- $("#mysidebar").navgoco({
- caretHtml: '',
- accordion: true,
- openClass: 'active', // open
- save: false, // leave false or nav highlighting doesn't work right
- cookie: {
- name: 'navgoco',
- expires: false,
- path: '/'
- },
- slide: {
- duration: 400,
- easing: 'swing'
- }
- });
-
- $("#collapseAll").click(function(e) {
- e.preventDefault();
- $("#mysidebar").navgoco('toggle', false);
- });
-
- $("#expandAll").click(function(e) {
- e.preventDefault();
- $("#mysidebar").navgoco('toggle', true);
- });
-
- });
-
- </script>
- <script>
- $(function () {
- $('[data-toggle="tooltip"]').tooltip()
- })
- </script>
-
-
-</head>
-<body>
-<!-- Navigation -->
-
-<nav class="navbar navbar-inverse navbar-fixed-top">
- <div class="container topnavlinks">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
- <img src="images/hudi_site_logo.png" alt="Hudi logo"/>
- <!--Hudi-->
- </span></a>
- </div>
- <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
- <ul class="nav navbar-nav navbar-right">
- <!-- entries without drop-downs appear here -->
-
-
-
- <li><a href="news">News</a></li>
-
-
-
- <li><a href="community.html">Community</a></li>
-
-
-
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
-
-
-
- <!-- entries with drop-downs appear here -->
- <!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
- <li><a href="dev_setup.html">Setup</a></li>
-
-
-
- <li><a href="api_docs.html">API Docs</a></li>
-
-
-
- <li><a href="code_and_design.html">Code Structure</a></li>
-
-
-
- <li class="dropdownActive"><a href="roadmap.html">Roadmap</a></li>
-
-
- </ul>
- </li>
-
-
-
- <li>
-
-
-
- <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Roadmap page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
-
-<li>
-
-
- <!--comment out this block if you want to hide search-->
- <li>
- <!--start search-->
- <div id="search-demo-container">
- <input type="text" id="search-input" placeholder="search...">
- <ul id="results-container"></ul>
- </div>
- <script src="js/jekyll-search.js" type="text/javascript"></script>
- <script type="text/javascript">
- SimpleJekyllSearch.init({
- searchInput: document.getElementById('search-input'),
- resultsContainer: document.getElementById('results-container'),
- dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}" title="Roadmap">{title}</a></li>',
- noResultsText: 'No results found.',
- limit: 10,
- fuzzy: true,
- })
- </script>
- <!--end search-->
- </li>
- </ul>
- </div>
- </div>
- <!-- /.container -->
-</nav>
-
-<!-- Page Content -->
-<div class="container">
- <div class="col-lg-12"> </div>
- <!-- Content Row -->
- <div class="row">
- <!-- Sidebar Column -->
- <div class="col-md-3">
-
-
-
-
-
-
-
-
-
-
-<ul id="mysidebar" class="nav">
- <li class="sidebarTitle">Latest Version</li>
-
-
-
- <li>
- <a href="#">Getting Started</a>
- <ul>
-
-
-
- <li><a href="index.html">Overview</a></li>
-
-
-
-
-
-
- <li><a href="quickstart.html">Quickstart</a></li>
-
-
-
-
-
-
- <li><a href="use_cases.html">Use Cases</a></li>
-
-
-
-
-
-
- <li><a href="powered_by.html">Talks & Powered By</a></li>
-
-
-
-
- </ul>
-
-
-
- <li>
- <a href="#">Documentation</a>
- <ul>
-
-
-
- <li><a href="concepts.html">Concepts</a></li>
-
-
-
-
-
-
- <li><a href="implementation.html">Implementation</a></li>
-
-
-
-
-
-
- <li><a href="configurations.html">Configurations</a></li>
-
-
-
-
-
-
- <li><a href="sql_queries.html">SQL Queries</a></li>
-
-
-
-
-
-
- <li><a href="migration_guide.html">Migration Guide</a></li>
-
-
-
-
-
-
- <li><a href="incremental_processing.html">Incremental Processing</a></li>
-
-
-
-
-
-
- <li><a href="admin_guide.html">Admin Guide</a></li>
-
-
-
-
-
-
- <li><a href="comparison.html">Comparison</a></li>
-
-
-
-
- </ul>
-
-
-
- <!-- if you aren't using the accordion, uncomment this block:
- <p class="external">
- <a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
- </p>
- -->
- </li>
-</ul>
-</div>
-
-<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
-<script>$("li.active").parents('li').toggleClass("active");</script>
-
- <!-- Content Column -->
- <div class="col-md-9">
- <div class="post-header">
- <h1 class="post-title-main">Roadmap</h1>
-</div>
-
-
-
-<div class="post-content">
-
-
-
-
-
-<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
-<script>
-$( document ).ready(function() {
- // Handler for .ready() called.
-
-$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
-
-/* this offset helps account for the space taken up by the floating toolbar. */
-$('#toc').on('click', 'a', function() {
- var target = $(this.getAttribute('href'))
- , scroll_target = target.offset().top
-
- $(window).scrollTop(scroll_target - 10);
- return false
-})
-
-});
-</script>
-
-<div id="toc"></div>
-
-
-
-
-
-
- <h2 id="planned-features">Planned Features</h2>
-
-<ul>
- <li>Support for Self Joins - As of now, you cannot incrementally consume the same table more than once, since the InputFormat does not understand the QueryPlan.</li>
- <li>Hudi Spark Datasource - Allows for reading and writing data back using Apache Spark natively (without falling back to InputFormat), which can be more performant</li>
- <li>Hudi Presto Connector - Allows for querying data managed by Hudi using Presto natively, which can again boost <a href="https://prestodb.io/docs/current/release/release-0.138.html">performance</a></li>
-</ul>
-
-
-
- <div class="tags">
-
- </div>
-
-
-
-</div>
-
-<hr class="shaded"/>
-
-<footer>
- <div class="row">
- <div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
- </div>
- </div>
-</footer>
-
-
- </div>
- <!-- /.row -->
-</div>
-<!-- /.container -->
- </div>
-
-</body>
-
-</html>
\ No newline at end of file
diff --git a/content/s3_hoodie.html b/content/s3_hoodie.html
index 7eadb35..217005c 100644
--- a/content/s3_hoodie.html
+++ b/content/s3_hoodie.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -428,9 +429,19 @@ export HOODIE_ENV_fs_DOT_s3n_DOT_impl=org.apache.hadoop.fs.s3a.S3AFileSystem
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/search.json b/content/search.json
index c77ec9b..3f7eb15 100644
--- a/content/search.json
+++ b/content/search.json
@@ -15,28 +15,6 @@
{
-"title": "API Docs",
-"tags": "",
-"keywords": "usecases",
-"url": "api_docs.html",
-"summary": ""
-}
-,
-
-
-
-{
-"title": "Code Structure",
-"tags": "",
-"keywords": "usecases",
-"url": "code_and_design.html",
-"summary": ""
-}
-,
-
-
-
-{
"title": "Community",
"tags": "",
"keywords": "usecases",
@@ -84,7 +62,7 @@
"title": "Developer Setup",
"tags": "",
"keywords": "developer setup",
-"url": "dev_setup.html",
+"url": "contributing.html",
"summary": ""
}
,
@@ -127,22 +105,22 @@
{
-"title": "Apache Hudi Overview",
+"title": "What is Hudi?",
"tags": "getting_started",
"keywords": "homepage",
"url": "index.html",
-"summary": "Hudi lowers data latency across the board, while simultaneously achieving orders of magnitude of efficiency over traditional batch processing."
+"summary": "Hudi brings stream processing to big data, providing fresh data while being an order of magnitude efficient over traditional batch processing."
}
,
{
-"title": "",
+"title": "Migration Guide",
"tags": "",
-"keywords": "",
+"keywords": "migration guide",
"url": "migration_guide.html",
-"summary": ""
+"summary": "In this page, we will discuss some available tools for migrating your existing dataset into a Hudi dataset"
}
,
@@ -193,10 +171,10 @@
{
-"title": "Quickstart",
-"tags": "quickstart",
-"keywords": "quickstart",
-"url": "quickstart.html",
+"title": "Privacy Policy",
+"tags": "",
+"keywords": "privacy",
+"url": "privacy.html",
"summary": ""
}
,
@@ -204,10 +182,10 @@
{
-"title": "Roadmap",
-"tags": "",
-"keywords": "usecases",
-"url": "roadmap.html",
+"title": "Quickstart",
+"tags": "quickstart",
+"keywords": "quickstart",
+"url": "quickstart.html",
"summary": ""
}
,
@@ -265,6 +243,16 @@
{
+"title": "Hudi entered Apache Incubator",
+"tags": "news",
+"keywords": "",
+"url": "/strata-talk.html",
+"summary": ""
+}
+,
+
+
+{
"title": "Connect with us at Strata San Jose March 2017",
"tags": "news",
"keywords": "",
diff --git a/content/sitemap.xml b/content/sitemap.xml
index fdf498e..14f34a7 100644
--- a/content/sitemap.xml
+++ b/content/sitemap.xml
@@ -7,26 +7,20 @@
</url>
-
-
-
-
-
<url>
- <loc>http://0.0.0.0:4000/admin_guide.html</loc>
+ <loc>http://0.0.0.0:4000/strata-talk.html</loc>
</url>
+
+
- <url>
- <loc>http://0.0.0.0:4000/api_docs.html</loc>
- </url>
<url>
- <loc>http://0.0.0.0:4000/code_and_design.html</loc>
+ <loc>http://0.0.0.0:4000/admin_guide.html</loc>
</url>
@@ -56,7 +50,7 @@
<url>
- <loc>http://0.0.0.0:4000/dev_setup.html</loc>
+ <loc>http://0.0.0.0:4000/contributing.html</loc>
</url>
@@ -118,13 +112,13 @@
<url>
- <loc>http://0.0.0.0:4000/quickstart.html</loc>
+ <loc>http://0.0.0.0:4000/privacy.html</loc>
</url>
<url>
- <loc>http://0.0.0.0:4000/roadmap.html</loc>
+ <loc>http://0.0.0.0:4000/quickstart.html</loc>
</url>
diff --git a/content/sql_queries.html b/content/sql_queries.html
index 079ce7f..6936191 100644
--- a/content/sql_queries.html
+++ b/content/sql_queries.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -393,11 +394,7 @@ reading Parquet files, just like queries on non-Hudi tables.</p>
<h2 id="presto">Presto</h2>
-<p>Presto requires a <a href="https://github.com/prestodb/presto/pull/7002">patch</a> (until the PR is merged) and the hoodie-hadoop-mr-bundle jar to be placed
-into <code class="highlighter-rouge"><presto_install>/plugin/hive-hadoop2/</code>.</p>
-
-<div class="bs-callout bs-callout-info">Get involved to improve this integration <a href="https://github.com/uber/hoodie/issues/81">here</a></div>
-
+<p>Presto requires the <code class="highlighter-rouge">hoodie-presto-bundle</code> jar to be placed into <code class="highlighter-rouge"><presto_install>/plugin/hive-hadoop2/</code>, across the installation.</p>
<div class="tags">
@@ -413,9 +410,19 @@ into <code class="highlighter-rouge"><presto_install>/plugin/hive-hadoop2/
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/strata-talk.html b/content/strata-talk.html
index 1af7fa4..13a8375 100644
--- a/content/strata-talk.html
+++ b/content/strata-talk.html
@@ -5,7 +5,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="keywords" content="news, ">
-<title>Connect with us at Strata San Jose March 2017 | Hudi</title>
+<title>Hudi entered Apache Incubator | Hudi</title>
<link rel="stylesheet" href="css/syntax.css">
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -157,7 +158,7 @@
- <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Connect with us at Strata San Jose March 2017 page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
+ <a class="email" title="Submit feedback" href="#" onclick="javascript:window.location='mailto:dev@hudi.apache.org?subject=Hudi Documentation feedback&body=I have some feedback about the Hudi entered Apache Incubator page: ' + window.location.href;"><i class="fa fa-envelope-o"></i> Feedback</a>
<li>
@@ -175,7 +176,7 @@
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
- searchResultTemplate: '<li><a href="{url}" title="Connect with us at Strata San Jose March 2017">{title}</a></li>',
+ searchResultTemplate: '<li><a href="{url}" title="Hudi entered Apache Incubator">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
@@ -328,8 +329,8 @@
<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
<header class="post-header">
- <h1 class="post-title" itemprop="name headline">Connect with us at Strata San Jose March 2017</h1>
- <p class="post-meta"><time datetime="2016-12-30T00:00:00+00:00" itemprop="datePublished">Dec 30, 2016</time> /
+ <h1 class="post-title" itemprop="name headline">Hudi entered Apache Incubator</h1>
+ <p class="post-meta"><time datetime="2019-01-18T00:00:00+00:00" itemprop="datePublished">Jan 18, 2019</time> /
@@ -347,8 +348,7 @@
- <p>We will be presenting Hoodie & general concepts around how incremental processing works at Uber.
-Catch our talk <strong>“Incremental Processing on Hadoop At Uber”</strong></p>
+ <p>In the coming weeks, we will be moving in our new home on the Apache Incubator.</p>
</div>
@@ -364,9 +364,19 @@ Catch our talk <strong>“Incremental Processing on Hadoop At Uber”</strong></
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/content/use_cases.html b/content/use_cases.html
index 32d840d..6df8c34 100644
--- a/content/use_cases.html
+++ b/content/use_cases.html
@@ -104,7 +104,8 @@
<a class="fa fa-lg navbar-brand" href="index.html"> <span class="projectTitle">
<img src="images/hudi_site_logo.png" alt="Hudi logo"/>
<!--Hudi-->
- </span></a>
+ </span><br/>
+ <p class="navbar-incubate">(Incubating)</p></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
@@ -120,7 +121,7 @@
- <li><a href="https://github.com/uber/hoodie" target="_blank">Github</a></li>
+ <li><a href="https://github.com/uber/hoodie" target="_blank">Code</a></li>
@@ -129,23 +130,23 @@
<li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developer Resources<b class="caret"></b></a>
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="dev_setup.html">Setup</a></li>
+ <li><a href="contributing.html">Contributing</a></li>
- <li><a href="api_docs.html">API Docs</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/HUDI" target="_blank">Wiki/Designs</a></li>
- <li><a href="code_and_design.html">Code Structure</a></li>
+ <li><a href="https://issues.apache.org/jira/projects/HUDI/summary" target="_blank">Issues</a></li>
- <li><a href="roadmap.html">Roadmap</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=HUDI" target="_blank">Blog</a></li>
</ul>
@@ -413,9 +414,19 @@ each run into a Hudi dataset, which can then be incrementally tailed (just like
<footer>
<div class="row">
<div class="col-lg-12 footer">
- ©2019 . All rights reserved. <br />
- Site last generated: Feb 16, 2019 <br />
- <!--<p><img src="images/company_logo.png" alt="Company logo"/></p>-->
+ <p>
+ Copyright © <span id="copyright-year">2019</span> <a href="https://apache.org">The Apache Software Foundation</a>,
+ Licensed under the Apache License, Version 2.0<br>
+ Apache and the Apache feather logo are trademarks of The Apache Software Foundation.| <a href="/privacy">Privacy Policy</a><br>
+ <a class="footer-link-img" href="https://apache.org">
+ <img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
+ </p>
+ <p>
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
+ stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
+ reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+ </p>
</div>
</div>
</footer>
diff --git a/docs/_includes/footer.html b/docs/_includes/footer.html
index c920c5c..ed02cf6 100755
--- a/docs/_includes/footer.html
+++ b/docs/_includes/footer.html
@@ -9,7 +9,7 @@
<img src="images/asf_logo.svg" alt="The Apache Software Foundation" height="100px" widh="50px"></a>
</p>
<p>
- Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of <a href="http://incubator.apache.org/">Apache Incubator</a>.
+ Apache Hudi is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the <a href="http://incubator.apache.org/">Apache Incubator</a>.
Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a
reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.