You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by gi...@apache.org on 2022/12/16 18:40:10 UTC
[iceberg-docs] branch asf-site updated: deploy: e48fc387b34a8c36022bf320de22a207b02c54bc
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/iceberg-docs.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 5174d7f9 deploy: e48fc387b34a8c36022bf320de22a207b02c54bc
5174d7f9 is described below
commit 5174d7f9302ddd2de11ccf3f1b09decc27771be6
Author: pvary <pv...@users.noreply.github.com>
AuthorDate: Fri Dec 16 18:40:05 2022 +0000
deploy: e48fc387b34a8c36022bf320de22a207b02c54bc
---
docs/1.0.0/api/index.html | 22 +++++++++++-----------
docs/1.0.0/custom-catalog/index.html | 12 ++++++------
docs/1.0.0/docssearch.json | 2 +-
docs/1.0.0/evolution/index.html | 2 +-
docs/1.0.0/flink/index.html | 4 ++--
docs/1.0.0/hive/index.html | 16 ++++++++++++----
docs/1.0.0/index.html | 2 +-
docs/1.0.0/index.xml | 4 ++--
docs/1.0.0/java-api-quickstart/index.html | 8 ++++----
docs/1.0.0/maintenance/index.html | 6 +++---
10 files changed, 43 insertions(+), 35 deletions(-)
diff --git a/docs/1.0.0/api/index.html b/docs/1.0.0/api/index.html
index 0f73a582..89766426 100644
--- a/docs/1.0.0/api/index.html
+++ b/docs/1.0.0/api/index.html
@@ -14,17 +14,17 @@
<i class="fa fa-chevron-down"></i></a></li><div id=Integrations class=collapse><ul class=sub-menu><li><a href=../aws/>AWS</a></li><li><a href=../dell/>Dell</a></li><li><a href=../jdbc/>JDBC</a></li><li><a href=../nessie/>Nessie</a></li></ul></div><li><a class=chevron-toggle data-toggle=collapse data-parent=full href=#API><span>API</span>
<i class="fa fa-chevron-right"></i>
<i class="fa fa-chevron-down"></i></a></li><div id=API class="collapse in"><ul class=sub-menu><li><a href=../java-api-quickstart/>Java Quickstart</a></li><li><a id=active href=../api/>Java API</a></li><li><a href=../custom-catalog/>Java Custom Catalog</a></li></ul></div><li><a href=https://iceberg.apache.org/docs/1.0.0/../../javadoc/latest><span>Javadoc</span></a></li></div></div><div id=content class=markdown-body><div class=margin-for-toc><h1 id=iceberg-java-api>Iceberg Java API</h1><h [...]
-</span></span></code></pre></div><p>To configure a scan, call <code>filter</code> and <code>select</code> on the <code>TableScan</code> to get a new <code>TableScan</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>TableScan filteredScan <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span [...]
+</span></span></code></pre></div><p>To configure a scan, call <code>filter</code> and <code>select</code> on the <code>TableScan</code> to get a new <code>TableScan</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>TableScan filteredScan <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span [...]
</span></span></code></pre></div><p>Calls to configuration methods create a new <code>TableScan</code> so that each <code>TableScan</code> is immutable and won’t change unexpectedly if shared across threads.</p><p>When a scan is configured, <code>planFiles</code>, <code>planTasks</code>, and <code>schema</code> are used to return files, tasks, and the read projection.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4; [...]
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>5</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>select</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>Schema projection <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>schema</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>Iterable<span style=color:#f92672><</span>CombinedScanTask<span style=color:#f92672>></span> tasks <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>planTasks</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>Use <code>asOfTime</code> or <code>useSnapshot</code> to configure the table snapshot for time travel queries.</p><h4 id=row-level>Row level</h4><p>Iceberg table scans start by creating a <code>ScanBuilder</code> object with <code>IcebergGenerics.read</code>.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><spa [...]
-</span></span></code></pre></div><p>To configure a scan, call <code>where</code> and <code>select</code> on the <code>ScanBuilder</code> to get a new <code>ScanBuilder</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>scanBuilder<span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=colo [...]
+</span></span></code></pre></div><p>To configure a scan, call <code>where</code> and <code>select</code> on the <code>ScanBuilder</code> to get a new <code>ScanBuilder</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>scanBuilder<span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=colo [...]
</span></span></code></pre></div><p>When a scan is configured, call method <code>build</code> to execute scan. <code>build</code> return <code>CloseableIterable<Record></code></p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>CloseableIterable<span style=color:#f92672><</span>Record<span style=color:#f92672>></span> result <span styl [...]
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>5</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>where <code>Record</code> is Iceberg record for iceberg-data module <code>org.apache.iceberg.data.Record</code>.</p><h3 id=update-operations>Update operations</h3><p><code>Table</code> also exposes operations that update the table. These operations use a builder pattern, <a href=../../../javadoc/1.0.0/index.html?org/apache/iceberg/PendingUpdate.html><code>PendingUpdate</code></a>, that commits when <code>PendingUpdate#commit</code> is called.</p><p>For [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>addColumn</span><span style=color:#f92672>(</span><span style=color:#e6db74>"count"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>LongType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>())</span>
@@ -39,23 +39,23 @@
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>t<span style=color:#f92672>.</span><span style=color:#a6e22e>commitTransaction</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><h2 id=types>Types</h2><p>Iceberg data types are located in the <a href=../../../javadoc/1.0.0/index.html?org/apache/iceberg/types/package-summary.html><code>org.apache.iceberg.types</code> package</a>.</p><h3 id=primitives>Primitives</h3><p>Primitive type instances are available from static methods in each type class. Types without parameters use <code>get</code>, and types like <code>decimal</code> use factory methods:</p><div class=highlight><pre tabin [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DoubleType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span> <span style=color:#75715e>// double
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DecimalType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span>9<span style=color:#f92672>,</span> 2<span style=color:#f92672>)</span> <span style=color:#75715e>// decimal(9, 2)
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DecimalType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span><span style=color:#ae81ff>9</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>2</span><span style=color:#f92672>)</span> <span style=color:#75715e>// decimal(9, 2)
</span></span></span></code></pre></div><h3 id=nested-types>Nested types</h3><p>Structs, maps, and lists are created using factory methods in type classes.</p><p>Like struct fields, map keys or values and list elements are tracked as nested fields. Nested fields track <a href=../evolution#correctness>field IDs</a> and nullability.</p><p>Struct fields are created using <code>NestedField.optional</code> or <code>NestedField.required</code>. Map value and list element nullability is set in [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>StructType struct <span style=color:#f92672>=</span> Struct<span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span>
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> <span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color: [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span>2<span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:# [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span><span style=color:#ae81ff>2</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color: [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>)</span>
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#75715e>// map<1 key: int, 2 value: optional string>
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>MapType map <span style=color:#f92672>=</span> MapType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofOptional</span><span style=color:#f92672>(</span>
-</span></span><span style=display:flex><span> 1<span style=color:#f92672>,</span> 2<span style=color:#f92672>,</span>
+</span></span><span style=display:flex><span> <span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>2</span><span style=color:#f92672>,</span>
</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>(),</span>
</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>)</span>
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#75715e>// array<1 element: int>
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span>ListType list <span style=color:#f92672>=</span> ListType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofRequired</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> IntegerType<span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>());</span>
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span>ListType list <span style=color:#f92672>=</span> ListType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofRequired</span><span style=color:#f92672>(</span><span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> IntegerType<span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>());</span>
</span></span></code></pre></div><h2 id=expressions>Expressions</h2><p>Iceberg’s expressions are used to configure table scans. To create expressions, use the factory methods in <a href=../../../javadoc/1.0.0/index.html?org/apache/iceberg/expressions/Expressions.html><code>Expressions</code></a>.</p><p>Supported predicate expressions are:</p><ul><li><code>isNull</code></li><li><code>notNull</code></li><li><code>equal</code></li><li><code>notEqual</code></li><li><code>lessThan</code [...]
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>greaterThanOrEqual</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> 10<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>greaterThanOrEqual</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>5</span><span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>10</span><span style=color:#f92672>))</span>
</span></span></code></pre></div><h2 id=modules>Modules</h2><p>Iceberg table support is organized in library modules:</p><ul><li><code>iceberg-common</code> contains utility classes used in other modules</li><li><code>iceberg-api</code> contains the public Iceberg API, including expressions, types, tables, and operations</li><li><code>iceberg-arrow</code> is an implementation of the Iceberg type system for reading and writing data stored in Iceberg tables using Apache Arrow as the in-mem [...]
<script src=https://iceberg.apache.org/docs/1.0.0//js/jquery.easing.min.js></script>
<script type=text/javascript src=https://iceberg.apache.org/docs/1.0.0//js/search.js></script>
diff --git a/docs/1.0.0/custom-catalog/index.html b/docs/1.0.0/custom-catalog/index.html
index 88181cdf..9cfa0a20 100644
--- a/docs/1.0.0/custom-catalog/index.html
+++ b/docs/1.0.0/custom-catalog/index.html
@@ -43,7 +43,7 @@
</span></span><span style=display:flex><span> String oldMetadataLocation <span style=color:#f92672>=</span> base<span style=color:#f92672>.</span><span style=color:#a6e22e>location</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Write new metadata using helper method
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span> String newMetadataLocation <span style=color:#f92672>=</span> writeNewMetadata<span style=color:#f92672>(</span>metadata<span style=color:#f92672>,</span> currentVersion<span style=color:#f92672>()</span> <span style=color:#f92672>+</span> 1<span style=color:#f92672>);</span>
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span> String newMetadataLocation <span style=color:#f92672>=</span> writeNewMetadata<span style=color:#f92672>(</span>metadata<span style=color:#f92672>,</span> currentVersion<span style=color:#f92672>()</span> <span style=color:#f92672>+</span> <span style=color:#ae81ff>1</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Example custom service which updates the metadata location for the given db and table atomically
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>updateMetadataLocation</span><span style=color:#f92672>(</span>dbName<span style=color:#f92672>,</span> tableName<span style=color:#f92672>,</span> oldMetadataLocation<span style=color:#f92672>,</span> newMetadataLocation<span style=color:#f92672>);</span>
@@ -75,7 +75,7 @@ See the next section about implementing and loading a custom catalog.</p><h3 id=
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>@Override</span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>protected</span> TableOperations <span style=color:#a6e22e>newTableOps</span><span style=color:#f92672>(</span>TableIdentifier tableIdentifier<span style=color:#f92672>)</span> <span style=color:#f92672>{</span>
-</span></span><span style=display:flex><span> String dbName <span style=color:#f92672>=</span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>);</span>
+</span></span><span style=display:flex><span> String dbName <span style=color:#f92672>=</span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span> String tableName <span style=color:#f92672>=</span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>name</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// instantiate the CustomTableOperations
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> <span style=color:#66d9ef>return</span> <span style=color:#66d9ef>new</span> CustomTableOperations<span style=color:#f92672>(</span>configuration<span style=color:#f92672>,</span> dbName<span style=color:#f92672>,</span> tableName<span style=color:#f92672>);</span>
@@ -94,22 +94,22 @@ See the next section about implementing and loading a custom catalog.</p><h3 id=
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>return</span> String<span style=color:#f92672>.</span><span style=color:#a6e22e>format</span><span style=color:#f92672>(</span>
</span></span><span style=display:flex><span> <span style=color:#e6db74>"%s/%s.db/%s"</span><span style=color:#f92672>,</span> tableLocation<span style=color:#f92672>,</span>
-</span></span><span style=display:flex><span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>levels</span><span style=color:#f92672>()[</span>0<span style=color:#f92672>],</span>
+</span></span><span style=display:flex><span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>levels</span><span style=color:#f92672>()[</span><span style=color:#ae81ff>0</span><span style=color:#f92672>],</span>
</span></span><span style=display:flex><span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>name</span><span style=color:#f92672>());</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>}</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>@Override</span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>public</span> <span style=color:#66d9ef>boolean</span> <span style=color:#a6e22e>dropTable</span><span style=color:#f92672>(</span>TableIdentifier identifier<span style=color:#f92672>,</span> <span style=color:#66d9ef>boolean</span> purge<span style=color:#f92672>)</span> <span style=color:#f92672>{</span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Example service to delete table
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>deleteTable</span><span style=color:#f92672>(</span>identifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>),</span> identifier<span style=color:#f92672>.</span><sp [...]
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>deleteTable</span><span style=color:#f92672>(</span>identifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>),</span> identifier<spa [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>}</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>@Override</span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>public</span> <span style=color:#66d9ef>void</span> <span style=color:#a6e22e>renameTable</span><span style=color:#f92672>(</span>TableIdentifier from<span style=color:#f92672>,</span> TableIdentifier to<span style=color:#f92672>)</span> <span style=color:#f92672>{</span>
-</span></span><span style=display:flex><span> Preconditions<span style=color:#f92672>.</span><span style=color:#a6e22e>checkArgument</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>).</span><span style=color:#a6e22e>equals</span><span style=color:#f92672>(</span>to<span style=color [...]
+</span></span><span style=display:flex><span> Preconditions<span style=color:#f92672>.</span><span style=color:#a6e22e>checkArgument</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>).</span><span style=color:#a6e22e>equals</span><span style=color:#f [...]
</span></span><span style=display:flex><span> <span style=color:#e6db74>"Cannot move table between databases"</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Example service to rename table
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>renameTable</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>),</span> from<span style=color:#f92672>.</span><span style=col [...]
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>renameTable</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>),</span> from<span style=colo [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>}</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// implement this method to read catalog name and properties during initialization
diff --git a/docs/1.0.0/docssearch.json b/docs/1.0.0/docssearch.json
index f640cbfa..2ca9961c 100644
--- a/docs/1.0.0/docssearch.json
+++ b/docs/1.0.0/docssearch.json
@@ -1 +1 @@
-[{"categories":null,"content":" Getting Started The latest version of Iceberg is 1.0.0.\nSpark is currently the most feature-rich compute engine for Iceberg operations. We recommend you to get started with Spark to understand Iceberg concepts and features with examples. You can also view documentations of using Iceberg with other compute engine under the Engines tab.\nUsing Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:\nspark-shell --packages org.apache.i [...]
\ No newline at end of file
+[{"categories":null,"content":" Getting Started The latest version of Iceberg is 1.0.0.\nSpark is currently the most feature-rich compute engine for Iceberg operations. We recommend you to get started with Spark to understand Iceberg concepts and features with examples. You can also view documentations of using Iceberg with other compute engine under the Engines tab.\nUsing Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:\nspark-shell --packages org.apache.i [...]
\ No newline at end of file
diff --git a/docs/1.0.0/evolution/index.html b/docs/1.0.0/evolution/index.html
index 5b60788b..46dc7c42 100644
--- a/docs/1.0.0/evolution/index.html
+++ b/docs/1.0.0/evolution/index.html
@@ -17,7 +17,7 @@
<em>The data for 2008 is partitioned by month. Starting from 2009 the table is updated so that the data is instead partitioned by day. Both partitioning layouts are able to coexist in the same table.</em></p><p>Iceberg uses <a href=../partitioning>hidden partitioning</a>, so you don’t <em>need</em> to write queries for a specific partition layout to be fast. Instead, you can write queries that select the data you need, and Iceberg automatically prunes out files that don’t con [...]
For example, the following code could be used to update the partition spec to add a new partition field that places <code>id</code> column values into 8 buckets and remove an existing partition field <code>category</code>:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>Table sampleTable <span style=color:#f92672>=</span> <span style=color: [...]
</span></span><span style=display:flex><span>sampleTable<span style=color:#f92672>.</span><span style=color:#a6e22e>updateSpec</span><span style=color:#f92672>()</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>addField</span><span style=color:#f92672>(</span>bucket<span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 8<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>addField</span><span style=color:#f92672>(</span>bucket<span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>8</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>removeField</span><span style=color:#f92672>(</span><span style=color:#e6db74>"category"</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>Spark supports updating partition spec through its <code>ALTER TABLE</code> SQL statement, see more details in <a href=../spark-ddl/#alter-table--add-partition-field>Spark SQL</a>.</p><h2 id=sort-order-evolution>Sort order evolution</h2><p>Similar to partition spec, Iceberg sort order can also be updated in an existing table.
diff --git a/docs/1.0.0/flink/index.html b/docs/1.0.0/flink/index.html
index 68477411..3c45cba9 100644
--- a/docs/1.0.0/flink/index.html
+++ b/docs/1.0.0/flink/index.html
@@ -222,7 +222,7 @@ For an unpartitioned iceberg table, its data will be completely overwritten by <
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>env</span><span style=color:#f92672>(</span>env<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>tableLoader</span><span style=color:#f92672>(</span>tableLoader<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>streaming</span><span style=color:#f92672>(</span><span style=color:#66d9ef>true</span><span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>startSnapshotId</span><span style=color:#f92672>(</span>3821550127947089987L<span style=color:#f92672>)</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>startSnapshotId</span><span style=color:#f92672>(</span><span style=color:#ae81ff>3821550127947089987L</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#75715e>// Print all records to stdout.
@@ -264,7 +264,7 @@ CDC read is not supported yet.</p><div class=highlight><pre tabindex=0 style=col
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>assignerFactory</span><span style=color:#f92672>(</span><span style=color:#66d9ef>new</span> SimpleSplitAssignerFactory<span style=color:#f92672>())</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>streaming</span><span style=color:#f92672>(</span><span style=color:#66d9ef>true</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>streamingStartingStrategy</span><span style=color:#f92672>(</span>StreamingStartingStrategy<span style=color:#f92672>.</span><span style=color:#a6e22e>INCREMENTAL_FROM_LATEST_SNAPSHOT</span><span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>monitorInterval</span><span style=color:#f92672>(</span>Duration<span style=color:#f92672>.</span><span style=color:#a6e22e>ofSeconds</span><span style=color:#f92672>(</span>60<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>monitorInterval</span><span style=color:#f92672>(</span>Duration<span style=color:#f92672>.</span><span style=color:#a6e22e>ofSeconds</span><span style=color:#f92672>(</span><span style=color:#ae81ff>60</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>DataStream<span style=color:#f92672><</span>RowData<span style=color:#f92672>></span> stream <span style=color:#f92672>=</span> env<span style=color:#f92672>.</span><span style=color:#a6e22e>fromSource</span><span style=color:#f92672>(</span>
diff --git a/docs/1.0.0/hive/index.html b/docs/1.0.0/hive/index.html
index 430123f8..28c995d6 100644
--- a/docs/1.0.0/hive/index.html
+++ b/docs/1.0.0/hive/index.html
@@ -14,7 +14,8 @@
<i class="fa fa-chevron-down"></i></a></li><div id=Integrations class=collapse><ul class=sub-menu><li><a href=../aws/>AWS</a></li><li><a href=../dell/>Dell</a></li><li><a href=../jdbc/>JDBC</a></li><li><a href=../nessie/>Nessie</a></li></ul></div><li><a class="chevron-toggle collapsed" data-toggle=collapse data-parent=full href=#API><span>API</span>
<i class="fa fa-chevron-right"></i>
<i class="fa fa-chevron-down"></i></a></li><div id=API class=collapse><ul class=sub-menu><li><a href=../java-api-quickstart/>Java Quickstart</a></li><li><a href=../api/>Java API</a></li><li><a href=../custom-catalog/>Java Custom Catalog</a></li></ul></div><li><a href=https://iceberg.apache.org/docs/1.0.0/../../javadoc/latest><span>Javadoc</span></a></li></div></div><div id=content class=markdown-body><div class=margin-for-toc><h1 id=hive>Hive</h1><p>Iceberg supports reading and writing I [...]
-a <a href=https://cwiki.apache.org/confluence/display/Hive/StorageHandlers>StorageHandler</a>.</p><h2 id=feature-support>Feature support</h2><p>Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features:</p><ul><li>Creating a table</li><li>Dropping a table</li><li>Reading a table</li><li>Inserting into a table (INSERT INTO)</li></ul><div class=warning>DML operations work only with MapReduce execution engine.</div><p>With Hive version 4.0.0-alpha-1 and above,
+a <a href=https://cwiki.apache.org/confluence/display/Hive/StorageHandlers>StorageHandler</a>.</p><h2 id=feature-support>Feature support</h2><p>Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features:</p><ul><li>Creating a table</li><li>Dropping a table</li><li>Reading a table</li><li>Inserting into a table (INSERT INTO)</li></ul><div class=warning>DML operations work only with MapReduce execution engine.</div><p>With Hive version 4.0.0-alpha-2 and above,
+the Iceberg integration when using HiveCatalog supports the following additional features:</p><ul><li>Altering a table with expiring snapshots.</li><li>Create a table like an existing table (CTLT table)</li><li>Support adding parquet compression type via Table properties <a href=https://spark.apache.org/docs/2.4.3/sql-data-sources-parquet.html#configuration>Compression types</a></li><li>Altering a table metadata location</li><li>Supporting table rollback</li><li>Honours sort orders on ex [...]
the Iceberg integration when using HiveCatalog supports the following additional features:</p><ul><li>Creating an Iceberg identity-partitioned table</li><li>Creating an Iceberg table with any partition spec, including the various transforms supported by Iceberg</li><li>Creating a table from an existing table (CTAS table)</li><li>Altering a table while keeping Iceberg and Hive schemas in sync</li><li>Altering the partition schema (updating columns)</li><li>Altering the partition schema by [...]
Hive’s classpath. These are provided by the <code>iceberg-hive-runtime</code> jar file. For example, if using the Hive shell, this
can be achieved by issuing a statement like so:</p><pre tabindex=0><code>add jar /path/to/iceberg-hive-runtime.jar;
@@ -61,12 +62,13 @@ The default is Parquet:</p><div class=highlight><pre tabindex=0 style=color:#f8f
</span></span></code></pre></div><h4 id=partitioned-tables>Partitioned tables</h4><p>You can create Iceberg partitioned tables using a command familiar to those who create non-Iceberg tables:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> x (i int) PARTITIONED <sp [...]
</span></span></code></pre></div><div class=info>The resulting table does not create partitions in HMS, but instead, converts partition data into Iceberg identity partitions.</div><p>Use the DESCRIBE command to get information about the Iceberg identity partitions:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>DESC [...]
</span></span></code></pre></div><p>The result is:</p><table><thead><tr><th>col_name</th><th>data_type</th><th>comment</th></tr></thead><tbody><tr><td>i</td><td>int</td><td></td></tr><tr><td>j</td><td>int</td><td></td></tr><tr><td></td><td>NULL</td><td>NULL</td></tr><tr><td># Partition Transform Information</td><td>NULL</td><td>NULL</td></tr><tr><td># col_name</td><td>transform_type</td><td>NULL</td></tr><tr><td>j</td><td>IDENTITY</td><td>NULL</td></tr></tbody></table><p>You can create I [...]
-(supported only in Hive 4.0.0-alpha-1):</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> x (i int, ts <span style=color:#66d9ef>timestamp</span>) PARTITIONED <span style=color:#66d9ef>BY</span> SPEC (<span style=color:#66d9ef>month</span>(ts), bucket(<span style=col [...]
+(supported only from Hive 4.0.0-alpha-1):</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> x (i int, ts <span style=color:#66d9ef>timestamp</span>) PARTITIONED <span style=color:#66d9ef>BY</span> SPEC (<span style=color:#66d9ef>month</span>(ts), bucket(<span style=c [...]
</span></span><span style=display:flex><span><span style=color:#66d9ef>DESCRIBE</span> x;
</span></span></code></pre></div><p>The result is:</p><table><thead><tr><th>col_name</th><th>data_type</th><th>comment</th></tr></thead><tbody><tr><td>i</td><td>int</td><td></td></tr><tr><td>ts</td><td>timestamp</td><td></td></tr><tr><td></td><td>NULL</td><td>NULL</td></tr><tr><td># Partition Transform Information</td><td>NULL</td><td>NULL</td></tr><tr><td># col_name</td><td>transform_type</td><td>NULL</td></tr><tr><td>ts</td><td>MONTH</td><td>NULL</td></tr><tr><td>i</td><td>BUCKET[2]</t [...]
The Iceberg table and the corresponding Hive table are created at the beginning of the query execution.
The data is inserted / committed when the query finishes. So for a transient period the table already exists but contains no data.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> target PARTITIONED <span style=color:#66d9ef>BY</span> SPEC (<span style=color:#66d9ef [...]
</span></span><span style=display:flex><span> <span style=color:#66d9ef>SELECT</span> <span style=color:#f92672>*</span> <span style=color:#66d9ef>FROM</span> <span style=color:#66d9ef>source</span>;
+</span></span></code></pre></div><h3 id=create-table-like-table>CREATE TABLE LIKE TABLE</h3><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> target <span style=color:#66d9ef>LIKE</span> <span style=color:#66d9ef>source</span> STORED <span style=color:#66d9ef>BY</span> [...]
</span></span></code></pre></div><h3 id=create-external-table-overlaying-an-existing-iceberg-table>CREATE EXTERNAL TABLE overlaying an existing Iceberg table</h3><p>The <code>CREATE EXTERNAL TABLE</code> command is used to overlay a Hive table “on top of” an existing Iceberg table. Iceberg
tables are created using either a <a href=../../../javadoc/1.0.0/index.html?org/apache/iceberg/catalog/Catalog.html><code>Catalog</code></a>, or an implementation of the <a href=../../../javadoc/1.0.0/index.html?org/apache/iceberg/Tables.html><code>Tables</code></a> interface, and Hive needs to be configured accordingly to
operate on these different types of table.</p><h4 id=hive-catalog-tables>Hive catalog tables</h4><p>As described before, tables created by the <code>HiveCatalog</code> with Hive engine feature enabled are directly visible by the
@@ -119,6 +121,8 @@ i.e. if columns are specified out-of-order an error will be thrown signalling th
</span></span></code></pre></div><p>During the migration the data files are not changed, only the appropriate Iceberg metadata files are created.
After the migration, handle the table as a normal Iceberg table.</p><h3 id=truncate-table>TRUNCATE TABLE</h3><p>The following command truncates the Iceberg table:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>TRUNCATE</span> <span style=color:#66d9ef>TABLE</span> t;
</span></span></code></pre></div><p>Using a partition specification is not allowed.</p><h3 id=drop-table>DROP TABLE</h3><p>Tables can be dropped using the <code>DROP TABLE</code> command:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>DROP</span> <span style=color:#66d9ef>TABLE</span> [<span style=color:#66d9ef>IF</ [...]
+</span></span></code></pre></div><h3 id=metadata-location>METADATA LOCATION</h3><p>The metadata location (snapshot location) only can be changed if the new path contains the exact same metadata json.
+It can be done only after migrating the table to Iceberg, the two operation cannot be done in one step.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>ALTER</span> <span style=color:#66d9ef>TABLE</span> t <span style=color:#66d9ef>set</span> TBLPROPERTIES (<span style=color:#e6db74>'metadata_location'</span> [...]
</span></span></code></pre></div><h2 id=dml-commands>DML Commands</h2><h3 id=select>SELECT</h3><p>Select statements work the same on Iceberg tables in Hive. You will see the Iceberg benefits over Hive in compilation and execution:</p><ul><li><strong>No file system listings</strong> - especially important on blob stores, like S3</li><li><strong>No partition listing from</strong> the Metastore</li><li><strong>Advanced partition filtering</strong> - the partition keys are not needed in the [...]
Also currently the statistics stored in the MetaStore are used for query planning. This is something we are planning to improve in the future.</p><h3 id=insert-into>INSERT INTO</h3><p>Hive supports the standard single-table INSERT INTO operation:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>INSERT</span> <span sty [...]
</span></span><span style=display:flex><span><span style=color:#66d9ef>VALUES</span> (<span style=color:#e6db74>'a'</span>, <span style=color:#ae81ff>1</span>);
@@ -140,10 +144,14 @@ To reference a metadata table the full name of the table should be used, like:
For these views it is possible to use projections / joins / filters / etc.
The function is available with the following syntax:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>SELECT</span> <span style=color:#f92672>*</span> <span style=color:#66d9ef>FROM</span> table_a <span style=color:#66d9ef>FOR</span> SYSTEM_TIME <span style=color:#66d9ef>AS</span> <span style=color:#66d9ef>OF</span> < [...]
</span></span><span style=display:flex><span><span style=color:#66d9ef>SELECT</span> <span style=color:#f92672>*</span> <span style=color:#66d9ef>FROM</span> table_a <span style=color:#66d9ef>FOR</span> SYSTEM_VERSION <span style=color:#66d9ef>AS</span> <span style=color:#66d9ef>OF</span> <span style=color:#ae81ff>1234567</span>;
-</span></span></code></pre></div><h2 id=type-compatibility>Type compatibility</h2><p>Hive and Iceberg support different set of types. Iceberg can perform type conversion automatically, but not for all
+</span></span></code></pre></div><p>You can expire snapshots of an Iceberg table using an ALTER TABLE query from Hive. You should periodically expire snapshots to delete data files that is no longer needed, and reduce the size of table metadata.</p><p>Each write to an Iceberg table from Hive creates a new snapshot, or version, of a table. Snapshots can be used for time-travel queries, or the table can be rolled back to any valid snapshot. Snapshots accumulate until they are expired by th [...]
+Enter a query to expire snapshots having the following timestamp: <code>2021-12-09 05:39:18.689000000</code></p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>ALTER</span> <span style=color:#66d9ef>TABLE</span> test_table <span style=color:#66d9ef>EXECUTE</span> expire_snapshots(<span style=color:#e6db74>'2021-12-0 [...]
+</span></span></code></pre></div><h3 id=type-compatibility>Type compatibility</h3><p>Hive and Iceberg support different set of types. Iceberg can perform type conversion automatically, but not for all
combinations, so you may want to understand the type conversion in Iceberg in prior to design the types of columns in
your tables. You can enable auto-conversion through Hadoop configuration (not enabled by default):</p><table><thead><tr><th>Config key</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td>iceberg.mr.schema.auto.conversion</td><td>false</td><td>if Hive should perform type auto-conversion</td></tr></tbody></table><h3 id=hive-type-to-iceberg-type>Hive type to Iceberg type</h3><p>This type conversion table describes how Hive types are converted to the Iceberg types. The conver [...]
-creating Iceberg table and writing to Iceberg table via Hive.</p><table><thead><tr><th>Hive</th><th>Iceberg</th><th>Notes</th></tr></thead><tbody><tr><td>boolean</td><td>boolean</td><td></td></tr><tr><td>short</td><td>integer</td><td>auto-conversion</td></tr><tr><td>byte</td><td>integer</td><td>auto-conversion</td></tr><tr><td>integer</td><td>integer</td><td></td></tr><tr><td>long</td><td>long</td><td></td></tr><tr><td>float</td><td>float</td><td></td></tr><tr><td>double</td><td>double</ [...]
+creating Iceberg table and writing to Iceberg table via Hive.</p><table><thead><tr><th>Hive</th><th>Iceberg</th><th>Notes</th></tr></thead><tbody><tr><td>boolean</td><td>boolean</td><td></td></tr><tr><td>short</td><td>integer</td><td>auto-conversion</td></tr><tr><td>byte</td><td>integer</td><td>auto-conversion</td></tr><tr><td>integer</td><td>integer</td><td></td></tr><tr><td>long</td><td>long</td><td></td></tr><tr><td>float</td><td>float</td><td></td></tr><tr><td>double</td><td>double</ [...]
+</span></span></code></pre></div><p>Rollback to a specific snapshot ID</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>ALTER</span> <span style=color:#66d9ef>TABLE</span> ice_t <span style=color:#66d9ef>EXECUTE</span> <span style=color:#66d9ef>ROLLBACK</span>(<span style=color:#ae81ff>1111</span>);
+</span></span></code></pre></div></div><div id=toc class=markdown-body><div id=full><nav id=TableOfContents><ul><li><a href=#feature-support>Feature support</a></li><li><a href=#enabling-iceberg-support-in-hive>Enabling Iceberg support in Hive</a><ul><li><a href=#hive-400-alpha-1>Hive 4.0.0-alpha-1</a></li><li><a href=#hive-23x-hive-31x>Hive 2.3.x, Hive 3.1.x</a></li></ul></li><li><a href=#catalog-management>Catalog Management</a><ul><li><a href=#global-hive-catalog>Global Hive catalog</ [...]
<script src=https://iceberg.apache.org/docs/1.0.0//js/jquery.easing.min.js></script>
<script type=text/javascript src=https://iceberg.apache.org/docs/1.0.0//js/search.js></script>
<script src=https://iceberg.apache.org/docs/1.0.0//js/bootstrap.min.js></script>
diff --git a/docs/1.0.0/index.html b/docs/1.0.0/index.html
index 9ccdbe0e..3dba0f85 100644
--- a/docs/1.0.0/index.html
+++ b/docs/1.0.0/index.html
@@ -1,4 +1,4 @@
-<!doctype html><html><head><meta name=generator content="Hugo 0.104.3"><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content><meta name=author content><title>Introduction</title><link href=./css/bootstrap.css rel=stylesheet><link href=./css/markdown.css rel=stylesheet><link href=./css/katex.min.css rel=stylesheet><link href=./css/iceberg-theme.css rel=stylesheet><link href=./f [...]
+<!doctype html><html><head><meta name=generator content="Hugo 0.108.0"><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content><meta name=author content><title>Introduction</title><link href=./css/bootstrap.css rel=stylesheet><link href=./css/markdown.css rel=stylesheet><link href=./css/katex.min.css rel=stylesheet><link href=./css/iceberg-theme.css rel=stylesheet><link href=./f [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span>
<span class=icon-bar></span>
diff --git a/docs/1.0.0/index.xml b/docs/1.0.0/index.xml
index ab96a01d..9437ce92 100644
--- a/docs/1.0.0/index.xml
+++ b/docs/1.0.0/index.xml
@@ -3,8 +3,8 @@ Spark is currently the most feature-rich compute engine for Iceberg operations.
Using Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:
spark-shell --packages org.</description></item><item><title>Hive</title><link>https://iceberg.apache.org/docs/1.0.0/hive/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/1.0.0/hive/</guid><description>Hive Iceberg supports reading and writing Iceberg tables through Hive by using a StorageHandler.
Feature support Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features:
-Creating a table Dropping a table Reading a table Inserting into a table (INSERT INTO) DML operations work only with MapReduce execution engine. With Hive version 4.0.0-alpha-1 and above, the Iceberg integration when using HiveCatalog supports the following additional features:
-Creating an Iceberg identity-partitioned table Creating an Iceberg table with any partition spec, including the various transforms supported by Iceberg Creating a table from an existing table (CTAS table) Altering a table while keeping Iceberg and Hive schemas in sync Altering the partition schema (updating columns) Altering the partition schema by specifying partition transforms Truncating a table Migrating tables in Avro, Parquet, or ORC (Non-ACID) format to Iceberg Reading the schema [...]
+Creating a table Dropping a table Reading a table Inserting into a table (INSERT INTO) DML operations work only with MapReduce execution engine. With Hive version 4.0.0-alpha-2 and above, the Iceberg integration when using HiveCatalog supports the following additional features:
+Altering a table with expiring snapshots.</description></item><item><title>AWS</title><link>https://iceberg.apache.org/docs/1.0.0/aws/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/1.0.0/aws/</guid><description>Iceberg AWS Integrations Iceberg provides integration with different AWS services through the iceberg-aws module. This section describes how to use Iceberg with AWS.
Enabling AWS Integration The iceberg-aws module is bundled with Spark and Flink engine runtimes for all versions from 0.11.0 onwards. However, the AWS clients are not bundled so that you can use the same client version as your application. You will need to provide the AWS v2 SDK because that is what Iceberg depends on.</description></item><item><title>Configuration</title><link>https://iceberg.apache.org/docs/1.0.0/configuration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><g [...]
Read properties Property Default Description read.split.target-size 134217728 (128 MB) Target size when combining data input splits read.split.metadata-target-size 33554432 (32 MB) Target size when combining metadata input splits read.split.planning-lookback 10 Number of bins to consider when combining input splits read.split.open-file-cost 4194304 (4 MB) The estimated cost to open a file, used as a minimum weight when combining splits.</description></item><item><title>Configuration</tit [...]
This creates an Iceberg catalog named hive_prod that loads tables from a Hive metastore:
diff --git a/docs/1.0.0/java-api-quickstart/index.html b/docs/1.0.0/java-api-quickstart/index.html
index cb611bfb..fe24e3e6 100644
--- a/docs/1.0.0/java-api-quickstart/index.html
+++ b/docs/1.0.0/java-api-quickstart/index.html
@@ -61,10 +61,10 @@ You can initialize a Hive catalog with a name and some properties.
</span></span><span style=display:flex><span><span style=color:#f92672>import</span> org.apache.iceberg.types.Types<span style=color:#f92672>;</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>Schema schema <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> Schema<span style=color:#f92672>(</span>
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> <span style=color:#e6db74>"level"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=co [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>2<span style=color:#f92672>,</span> <span style=color:#e6db74>"event_time"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>TimestampType</span><span style=color:#f92672>.</span><span [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>3<span style=color:#f92672>,</span> <span style=color:#e6db74>"message"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style= [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span>4<span style=color:#f92672>,</span> <span style=color:#e6db74>"call_stack"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>ListType</span><span style=color:#f92672>.</span><span style [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"level"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=col [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>2</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"event_time"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>TimestampType</span><span s [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>3</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"message"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=c [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span><span style=color:#ae81ff>4</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"call_stack"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>ListType</span><span style= [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>);</span>
</span></span></code></pre></div><p>When using the Iceberg API directly, type IDs are required. Conversions from other schema formats, like Spark, Avro, and Parquet will automatically assign new IDs.</p><p>When a table is created, all IDs in the schema are re-assigned to ensure uniqueness.</p><h3 id=convert-a-schema-from-avro>Convert a schema from Avro</h3><p>To create an Iceberg schema from an existing Avro schema, use converters in <code>AvroSchemaUtil</code>:</p><div class=highlight>< [...]
</span></span><span style=display:flex><span><span style=color:#f92672>import</span> org.apache.avro.Schema.Parser<span style=color:#f92672>;</span>
diff --git a/docs/1.0.0/maintenance/index.html b/docs/1.0.0/maintenance/index.html
index 72c9682b..d00ec82d 100644
--- a/docs/1.0.0/maintenance/index.html
+++ b/docs/1.0.0/maintenance/index.html
@@ -14,7 +14,7 @@
<i class="fa fa-chevron-down"></i></a></li><div id=Integrations class=collapse><ul class=sub-menu><li><a href=../aws/>AWS</a></li><li><a href=../dell/>Dell</a></li><li><a href=../jdbc/>JDBC</a></li><li><a href=../nessie/>Nessie</a></li></ul></div><li><a class="chevron-toggle collapsed" data-toggle=collapse data-parent=full href=#API><span>API</span>
<i class="fa fa-chevron-right"></i>
<i class="fa fa-chevron-down"></i></a></li><div id=API class=collapse><ul class=sub-menu><li><a href=../java-api-quickstart/>Java Quickstart</a></li><li><a href=../api/>Java API</a></li><li><a href=../custom-catalog/>Java Custom Catalog</a></li></ul></div><li><a href=https://iceberg.apache.org/docs/1.0.0/../../javadoc/latest><span>Javadoc</span></a></li></div></div><div id=content class=markdown-body><div class=margin-for-toc><h1 id=maintenance>Maintenance</h1><div class=info>Maintenance [...]
-</span></span><span style=display:flex><span><span style=color:#66d9ef>long</span> tsToExpire <span style=color:#f92672>=</span> System<span style=color:#f92672>.</span><span style=color:#a6e22e>currentTimeMillis</span><span style=color:#f92672>()</span> <span style=color:#f92672>-</span> <span style=color:#f92672>(</span>1000 <span style=color:#f92672>*</span> 60 <span style=color:#f92672>*</span> 60 <span style=color:#f92672>*</span> 24<span style=color:#f92672>);</span> <span style=co [...]
+</span></span><span style=display:flex><span><span style=color:#66d9ef>long</span> tsToExpire <span style=color:#f92672>=</span> System<span style=color:#f92672>.</span><span style=color:#a6e22e>currentTimeMillis</span><span style=color:#f92672>()</span> <span style=color:#f92672>-</span> <span style=color:#f92672>(</span><span style=color:#ae81ff>1000</span> <span style=color:#f92672>*</span> <span style=color:#ae81ff>60</span> <span style=color:#f92672>*</span> <span style=color:#ae81f [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>table<span style=color:#f92672>.</span><span style=color:#a6e22e>expireSnapshots</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>expireOlderThan</span><span style=color:#f92672>(</span>tsToExpire<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span>
@@ -40,13 +40,13 @@ FileSystem API to avoid unintentional deletion.</div><h2 id=optional-maintenance
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteDataFiles</span><span style=color:#f92672>(</span>table<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"date"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"2020-08-18"</span><span style=color:#f92672>))</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>option</span><span style=color:#f92672>(</span><span style=color:#e6db74>"target-file-size-bytes"</span><span style=color:#f92672>,</span> Long<span style=color:#f92672>.</span><span style=color:#a6e22e>toString</span><span style=color:#f92672>(</span>500 <span style=color:#f92672>*</span> 1024 <span style=color:#f92672>*</span> 1024<span style=color:#f92672>))</span> <spa [...]
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>option</span><span style=color:#f92672>(</span><span style=color:#e6db74>"target-file-size-bytes"</span><span style=color:#f92672>,</span> Long<span style=color:#f92672>.</span><span style=color:#a6e22e>toString</span><span style=color:#f92672>(</span><span style=color:#ae81ff>500</span> <span style=color:#f92672>*</span> <span style=color:#ae81ff>1024</span> <span style=c [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> <span style=color:#f92672>.</span><span style=color:#a6e22e>execute</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>The <code>files</code> metadata table is useful for inspecting data file sizes and determining when to compact partitions.</p><p>See the <a href=../../../javadoc/1.0.0/org/apache/iceberg/actions/RewriteDataFiles.html><code>RewriteDataFiles</code> Javadoc</a> to see more configuration options.</p><h3 id=rewrite-manifests>Rewrite manifests</h3><p>Iceberg uses metadata in its manifest list and manifest files speed up query planning and to prune unnecessar [...]
</span></span><span style=display:flex><span>SparkActions
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteManifests</span><span style=color:#f92672>(</span>table<span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteIf</span><span style=color:#f92672>(</span>file <span style=color:#f92672>-></span> file<span style=color:#f92672>.</span><span style=color:#a6e22e>length</span><span style=color:#f92672>()</span> <span style=color:#f92672><</span> 10 <span style=color:#f92672>*</span> 1024 <span style=color:#f92672>*</span> 1024<span style=color:#f92672>)</span> <span style=color:#75 [...]
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteIf</span><span style=color:#f92672>(</span>file <span style=color:#f92672>-></span> file<span style=color:#f92672>.</span><span style=color:#a6e22e>length</span><span style=color:#f92672>()</span> <span style=color:#f92672><</span> <span style=color:#ae81ff>10</span> <span style=color:#f92672>*</span> <span style=color:#ae81ff>1024</span> <span style=color:#f92672>*</ [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> <span style=color:#f92672>.</span><span style=color:#a6e22e>execute</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>See the <a href=../../../javadoc/1.0.0/org/apache/iceberg/actions/RewriteManifests.html><code>RewriteManifests</code> Javadoc</a> to see more configuration options.</p></div><div id=toc class=markdown-body><div id=full><nav id=TableOfContents><ul><li><a href=#recommended-maintenance>Recommended Maintenance</a><ul><li><a href=#expire-snapshots>Expire Snapshots</a></li><li><a href=#remove-old-metadata-files>Remove old metadata files</a></li><li><a href=# [...]
<script src=https://iceberg.apache.org/docs/1.0.0//js/jquery.easing.min.js></script>