You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by gi...@apache.org on 2022/12/16 18:19:43 UTC
[iceberg-docs] branch asf-site updated: deploy: 90f16bd2482d2e5fe061c85a798d639962285b65
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/iceberg-docs.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6fd39f81 deploy: 90f16bd2482d2e5fe061c85a798d639962285b65
6fd39f81 is described below
commit 6fd39f81f4a2921e90f1c780c17cc0d6069a26c9
Author: pvary <pv...@users.noreply.github.com>
AuthorDate: Fri Dec 16 18:19:38 2022 +0000
deploy: 90f16bd2482d2e5fe061c85a798d639962285b65
---
docs/0.14.1/api/index.html | 22 +++++++++++-----------
docs/0.14.1/custom-catalog/index.html | 12 ++++++------
docs/0.14.1/docssearch.json | 2 +-
docs/0.14.1/evolution/index.html | 2 +-
docs/0.14.1/flink/index.html | 4 ++--
docs/0.14.1/hive/index.html | 16 ++++++++++++----
docs/0.14.1/index.html | 2 +-
docs/0.14.1/index.xml | 4 ++--
docs/0.14.1/java-api-quickstart/index.html | 8 ++++----
docs/0.14.1/maintenance/index.html | 6 +++---
10 files changed, 43 insertions(+), 35 deletions(-)
diff --git a/docs/0.14.1/api/index.html b/docs/0.14.1/api/index.html
index fee5c3ac..02f6376a 100644
--- a/docs/0.14.1/api/index.html
+++ b/docs/0.14.1/api/index.html
@@ -14,17 +14,17 @@
<i class="fa fa-chevron-down"></i></a></li><div id=Integrations class=collapse><ul class=sub-menu><li><a href=../aws/>AWS</a></li><li><a href=../dell/>Dell</a></li><li><a href=../jdbc/>JDBC</a></li><li><a href=../nessie/>Nessie</a></li></ul></div><li><a class=chevron-toggle data-toggle=collapse data-parent=full href=#API><span>API</span>
<i class="fa fa-chevron-right"></i>
<i class="fa fa-chevron-down"></i></a></li><div id=API class="collapse in"><ul class=sub-menu><li><a href=../java-api-quickstart/>Java Quickstart</a></li><li><a id=active href=../api/>Java API</a></li><li><a href=../custom-catalog/>Java Custom Catalog</a></li><li><a href=../python-quickstart/>Python Quickstart</a></li><li><a href=../python-api-intro/>Python API</a></li><li><a href=../python-feature-support/>Python Feature Support</a></li></ul></div><li><a href=https://iceberg.apache.org/ [...]
-</span></span></code></pre></div><p>To configure a scan, call <code>filter</code> and <code>select</code> on the <code>TableScan</code> to get a new <code>TableScan</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>TableScan filteredScan <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span [...]
+</span></span></code></pre></div><p>To configure a scan, call <code>filter</code> and <code>select</code> on the <code>TableScan</code> to get a new <code>TableScan</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>TableScan filteredScan <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span [...]
</span></span></code></pre></div><p>Calls to configuration methods create a new <code>TableScan</code> so that each <code>TableScan</code> is immutable and won’t change unexpectedly if shared across threads.</p><p>When a scan is configured, <code>planFiles</code>, <code>planTasks</code>, and <code>schema</code> are used to return files, tasks, and the read projection.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4; [...]
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>5</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>select</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>Schema projection <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>schema</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>Iterable<span style=color:#f92672><</span>CombinedScanTask<span style=color:#f92672>></span> tasks <span style=color:#f92672>=</span> scan<span style=color:#f92672>.</span><span style=color:#a6e22e>planTasks</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>Use <code>asOfTime</code> or <code>useSnapshot</code> to configure the table snapshot for time travel queries.</p><h4 id=row-level>Row level</h4><p>Iceberg table scans start by creating a <code>ScanBuilder</code> object with <code>IcebergGenerics.read</code>.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><spa [...]
-</span></span></code></pre></div><p>To configure a scan, call <code>where</code> and <code>select</code> on the <code>ScanBuilder</code> to get a new <code>ScanBuilder</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>scanBuilder<span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=colo [...]
+</span></span></code></pre></div><p>To configure a scan, call <code>where</code> and <code>select</code> on the <code>ScanBuilder</code> to get a new <code>ScanBuilder</code> with those changes.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>scanBuilder<span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=colo [...]
</span></span></code></pre></div><p>When a scan is configured, call method <code>build</code> to execute scan. <code>build</code> return <code>CloseableIterable<Record></code></p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>CloseableIterable<span style=color:#f92672><</span>Record<span style=color:#f92672>></span> result <span styl [...]
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>where</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>5</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>where <code>Record</code> is Iceberg record for iceberg-data module <code>org.apache.iceberg.data.Record</code>.</p><h3 id=update-operations>Update operations</h3><p><code>Table</code> also exposes operations that update the table. These operations use a builder pattern, <a href=../../../javadoc/0.14.1/index.html?org/apache/iceberg/PendingUpdate.html><code>PendingUpdate</code></a>, that commits when <code>PendingUpdate#commit</code> is called.</p><p>Fo [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>addColumn</span><span style=color:#f92672>(</span><span style=color:#e6db74>"count"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>LongType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>())</span>
@@ -39,23 +39,23 @@
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>t<span style=color:#f92672>.</span><span style=color:#a6e22e>commitTransaction</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><h2 id=types>Types</h2><p>Iceberg data types are located in the <a href=../../../javadoc/0.14.1/index.html?org/apache/iceberg/types/package-summary.html><code>org.apache.iceberg.types</code> package</a>.</p><h3 id=primitives>Primitives</h3><p>Primitive type instances are available from static methods in each type class. Types without parameters use <code>get</code>, and types like <code>decimal</code> use factory methods:</p><div class=highlight><pre tabi [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DoubleType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span> <span style=color:#75715e>// double
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DecimalType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span>9<span style=color:#f92672>,</span> 2<span style=color:#f92672>)</span> <span style=color:#75715e>// decimal(9, 2)
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span>Types<span style=color:#f92672>.</span><span style=color:#a6e22e>DecimalType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span><span style=color:#ae81ff>9</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>2</span><span style=color:#f92672>)</span> <span style=color:#75715e>// decimal(9, 2)
</span></span></span></code></pre></div><h3 id=nested-types>Nested types</h3><p>Structs, maps, and lists are created using factory methods in type classes.</p><p>Like struct fields, map keys or values and list elements are tracked as nested fields. Nested fields track <a href=../evolution#correctness>field IDs</a> and nullability.</p><p>Struct fields are created using <code>NestedField.optional</code> or <code>NestedField.required</code>. Map value and list element nullability is set in [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>StructType struct <span style=color:#f92672>=</span> Struct<span style=color:#f92672>.</span><span style=color:#a6e22e>of</span><span style=color:#f92672>(</span>
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> <span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color: [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span>2<span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:# [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span><span style=color:#ae81ff>2</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"data"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color: [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>)</span>
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#75715e>// map<1 key: int, 2 value: optional string>
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>MapType map <span style=color:#f92672>=</span> MapType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofOptional</span><span style=color:#f92672>(</span>
-</span></span><span style=display:flex><span> 1<span style=color:#f92672>,</span> 2<span style=color:#f92672>,</span>
+</span></span><span style=display:flex><span> <span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>2</span><span style=color:#f92672>,</span>
</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>IntegerType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>(),</span>
</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>)</span>
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#75715e>// array<1 element: int>
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span>ListType list <span style=color:#f92672>=</span> ListType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofRequired</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> IntegerType<span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>());</span>
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span>ListType list <span style=color:#f92672>=</span> ListType<span style=color:#f92672>.</span><span style=color:#a6e22e>ofRequired</span><span style=color:#f92672>(</span><span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> IntegerType<span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>());</span>
</span></span></code></pre></div><h2 id=expressions>Expressions</h2><p>Iceberg’s expressions are used to configure table scans. To create expressions, use the factory methods in <a href=../../../javadoc/0.14.1/index.html?org/apache/iceberg/expressions/Expressions.html><code>Expressions</code></a>.</p><p>Supported predicate expressions are:</p><ul><li><code>isNull</code></li><li><code>notNull</code></li><li><code>equal</code></li><li><code>notEqual</code></li><li><code>lessThan</cod [...]
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>greaterThanOrEqual</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> 5<span style=color:#f92672>))</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> 10<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>greaterThanOrEqual</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>5</span><span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>lessThan</span><span style=color:#f92672>(</span><span style=color:#e6db74>"x"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>10</span><span style=color:#f92672>))</span>
</span></span></code></pre></div><h2 id=modules>Modules</h2><p>Iceberg table support is organized in library modules:</p><ul><li><code>iceberg-common</code> contains utility classes used in other modules</li><li><code>iceberg-api</code> contains the public Iceberg API, including expressions, types, tables, and operations</li><li><code>iceberg-arrow</code> is an implementation of the Iceberg type system for reading and writing data stored in Iceberg tables using Apache Arrow as the in-mem [...]
<script src=https://iceberg.apache.org/docs/0.14.1//js/jquery.easing.min.js></script>
<script type=text/javascript src=https://iceberg.apache.org/docs/0.14.1//js/search.js></script>
diff --git a/docs/0.14.1/custom-catalog/index.html b/docs/0.14.1/custom-catalog/index.html
index 172271e5..53bb5321 100644
--- a/docs/0.14.1/custom-catalog/index.html
+++ b/docs/0.14.1/custom-catalog/index.html
@@ -43,7 +43,7 @@
</span></span><span style=display:flex><span> String oldMetadataLocation <span style=color:#f92672>=</span> base<span style=color:#f92672>.</span><span style=color:#a6e22e>location</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Write new metadata using helper method
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span> String newMetadataLocation <span style=color:#f92672>=</span> writeNewMetadata<span style=color:#f92672>(</span>metadata<span style=color:#f92672>,</span> currentVersion<span style=color:#f92672>()</span> <span style=color:#f92672>+</span> 1<span style=color:#f92672>);</span>
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span> String newMetadataLocation <span style=color:#f92672>=</span> writeNewMetadata<span style=color:#f92672>(</span>metadata<span style=color:#f92672>,</span> currentVersion<span style=color:#f92672>()</span> <span style=color:#f92672>+</span> <span style=color:#ae81ff>1</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Example custom service which updates the metadata location for the given db and table atomically
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>updateMetadataLocation</span><span style=color:#f92672>(</span>dbName<span style=color:#f92672>,</span> tableName<span style=color:#f92672>,</span> oldMetadataLocation<span style=color:#f92672>,</span> newMetadataLocation<span style=color:#f92672>);</span>
@@ -75,7 +75,7 @@ See the next section about implementing and loading a custom catalog.</p><h3 id=
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>@Override</span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>protected</span> TableOperations <span style=color:#a6e22e>newTableOps</span><span style=color:#f92672>(</span>TableIdentifier tableIdentifier<span style=color:#f92672>)</span> <span style=color:#f92672>{</span>
-</span></span><span style=display:flex><span> String dbName <span style=color:#f92672>=</span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>);</span>
+</span></span><span style=display:flex><span> String dbName <span style=color:#f92672>=</span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span> String tableName <span style=color:#f92672>=</span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>name</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// instantiate the CustomTableOperations
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> <span style=color:#66d9ef>return</span> <span style=color:#66d9ef>new</span> CustomTableOperations<span style=color:#f92672>(</span>configuration<span style=color:#f92672>,</span> dbName<span style=color:#f92672>,</span> tableName<span style=color:#f92672>);</span>
@@ -94,22 +94,22 @@ See the next section about implementing and loading a custom catalog.</p><h3 id=
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>return</span> String<span style=color:#f92672>.</span><span style=color:#a6e22e>format</span><span style=color:#f92672>(</span>
</span></span><span style=display:flex><span> <span style=color:#e6db74>"%s/%s.db/%s"</span><span style=color:#f92672>,</span> tableLocation<span style=color:#f92672>,</span>
-</span></span><span style=display:flex><span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>levels</span><span style=color:#f92672>()[</span>0<span style=color:#f92672>],</span>
+</span></span><span style=display:flex><span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>levels</span><span style=color:#f92672>()[</span><span style=color:#ae81ff>0</span><span style=color:#f92672>],</span>
</span></span><span style=display:flex><span> tableIdentifier<span style=color:#f92672>.</span><span style=color:#a6e22e>name</span><span style=color:#f92672>());</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>}</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>@Override</span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>public</span> <span style=color:#66d9ef>boolean</span> <span style=color:#a6e22e>dropTable</span><span style=color:#f92672>(</span>TableIdentifier identifier<span style=color:#f92672>,</span> <span style=color:#66d9ef>boolean</span> purge<span style=color:#f92672>)</span> <span style=color:#f92672>{</span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Example service to delete table
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>deleteTable</span><span style=color:#f92672>(</span>identifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>),</span> identifier<span style=color:#f92672>.</span><sp [...]
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>deleteTable</span><span style=color:#f92672>(</span>identifier<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>),</span> identifier<spa [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>}</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>@Override</span>
</span></span><span style=display:flex><span> <span style=color:#66d9ef>public</span> <span style=color:#66d9ef>void</span> <span style=color:#a6e22e>renameTable</span><span style=color:#f92672>(</span>TableIdentifier from<span style=color:#f92672>,</span> TableIdentifier to<span style=color:#f92672>)</span> <span style=color:#f92672>{</span>
-</span></span><span style=display:flex><span> Preconditions<span style=color:#f92672>.</span><span style=color:#a6e22e>checkArgument</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>).</span><span style=color:#a6e22e>equals</span><span style=color:#f92672>(</span>to<span style=color [...]
+</span></span><span style=display:flex><span> Preconditions<span style=color:#f92672>.</span><span style=color:#a6e22e>checkArgument</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namespace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>).</span><span style=color:#a6e22e>equals</span><span style=color:#f [...]
</span></span><span style=display:flex><span> <span style=color:#e6db74>"Cannot move table between databases"</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// Example service to rename table
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>renameTable</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span>0<span style=color:#f92672>),</span> from<span style=color:#f92672>.</span><span style=col [...]
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span> CustomService<span style=color:#f92672>.</span><span style=color:#a6e22e>renameTable</span><span style=color:#f92672>(</span>from<span style=color:#f92672>.</span><span style=color:#a6e22e>namepsace</span><span style=color:#f92672>().</span><span style=color:#a6e22e>level</span><span style=color:#f92672>(</span><span style=color:#ae81ff>0</span><span style=color:#f92672>),</span> from<span style=colo [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>}</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> <span style=color:#75715e>// implement this method to read catalog name and properties during initialization
diff --git a/docs/0.14.1/docssearch.json b/docs/0.14.1/docssearch.json
index 8a0cdb53..5c6fc70c 100644
--- a/docs/0.14.1/docssearch.json
+++ b/docs/0.14.1/docssearch.json
@@ -1 +1 @@
-[{"categories":null,"content":" Getting Started The latest version of Iceberg is 0.14.1.\nSpark is currently the most feature-rich compute engine for Iceberg operations. We recommend you to get started with Spark to understand Iceberg concepts and features with examples. You can also view documentations of using Iceberg with other compute engine under the Engines tab.\nUsing Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:\nspark-shell --packages org.apache. [...]
\ No newline at end of file
+[{"categories":null,"content":" Getting Started The latest version of Iceberg is 0.14.1.\nSpark is currently the most feature-rich compute engine for Iceberg operations. We recommend you to get started with Spark to understand Iceberg concepts and features with examples. You can also view documentations of using Iceberg with other compute engine under the Engines tab.\nUsing Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:\nspark-shell --packages org.apache. [...]
\ No newline at end of file
diff --git a/docs/0.14.1/evolution/index.html b/docs/0.14.1/evolution/index.html
index 38f72d3b..3c0bad38 100644
--- a/docs/0.14.1/evolution/index.html
+++ b/docs/0.14.1/evolution/index.html
@@ -17,7 +17,7 @@
<em>The data for 2008 is partitioned by month. Starting from 2009 the table is updated so that the data is instead partitioned by day. Both partitioning layouts are able to coexist in the same table.</em></p><p>Iceberg uses <a href=../partitioning>hidden partitioning</a>, so you don’t <em>need</em> to write queries for a specific partition layout to be fast. Instead, you can write queries that select the data you need, and Iceberg automatically prunes out files that don’t con [...]
For example, the following code could be used to update the partition spec to add a new partition field that places <code>id</code> column values into 8 buckets and remove an existing partition field <code>category</code>:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>Table sampleTable <span style=color:#f92672>=</span> <span style=color: [...]
</span></span><span style=display:flex><span>sampleTable<span style=color:#f92672>.</span><span style=color:#a6e22e>updateSpec</span><span style=color:#f92672>()</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>addField</span><span style=color:#f92672>(</span>bucket<span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> 8<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>addField</span><span style=color:#f92672>(</span>bucket<span style=color:#f92672>(</span><span style=color:#e6db74>"id"</span><span style=color:#f92672>,</span> <span style=color:#ae81ff>8</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>removeField</span><span style=color:#f92672>(</span><span style=color:#e6db74>"category"</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>Spark supports updating partition spec through its <code>ALTER TABLE</code> SQL statement, see more details in <a href=../spark-ddl/#alter-table--add-partition-field>Spark SQL</a>.</p><h2 id=sort-order-evolution>Sort order evolution</h2><p>Similar to partition spec, Iceberg sort order can also be updated in an existing table.
diff --git a/docs/0.14.1/flink/index.html b/docs/0.14.1/flink/index.html
index 7218507d..0b73aa05 100644
--- a/docs/0.14.1/flink/index.html
+++ b/docs/0.14.1/flink/index.html
@@ -222,7 +222,7 @@ For an unpartitioned iceberg table, its data will be completely overwritten by <
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>env</span><span style=color:#f92672>(</span>env<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>tableLoader</span><span style=color:#f92672>(</span>tableLoader<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>streaming</span><span style=color:#f92672>(</span><span style=color:#66d9ef>true</span><span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>startSnapshotId</span><span style=color:#f92672>(</span>3821550127947089987L<span style=color:#f92672>)</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>startSnapshotId</span><span style=color:#f92672>(</span><span style=color:#ae81ff>3821550127947089987L</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#75715e>// Print all records to stdout.
@@ -264,7 +264,7 @@ CDC read is not supported yet.</p><div class=highlight><pre tabindex=0 style=col
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>assignerFactory</span><span style=color:#f92672>(</span><span style=color:#66d9ef>new</span> SimpleSplitAssignerFactory<span style=color:#f92672>())</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>streaming</span><span style=color:#f92672>(</span><span style=color:#66d9ef>true</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>streamingStartingStrategy</span><span style=color:#f92672>(</span>StreamingStartingStrategy<span style=color:#f92672>.</span><span style=color:#a6e22e>INCREMENTAL_FROM_LATEST_SNAPSHOT</span><span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>monitorInterval</span><span style=color:#f92672>(</span>Duration<span style=color:#f92672>.</span><span style=color:#a6e22e>ofSeconds</span><span style=color:#f92672>(</span>60<span style=color:#f92672>))</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>monitorInterval</span><span style=color:#f92672>(</span>Duration<span style=color:#f92672>.</span><span style=color:#a6e22e>ofSeconds</span><span style=color:#f92672>(</span><span style=color:#ae81ff>60</span><span style=color:#f92672>))</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>build</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>DataStream<span style=color:#f92672><</span>RowData<span style=color:#f92672>></span> stream <span style=color:#f92672>=</span> env<span style=color:#f92672>.</span><span style=color:#a6e22e>fromSource</span><span style=color:#f92672>(</span>
diff --git a/docs/0.14.1/hive/index.html b/docs/0.14.1/hive/index.html
index 17efe804..f0fb4aa1 100644
--- a/docs/0.14.1/hive/index.html
+++ b/docs/0.14.1/hive/index.html
@@ -14,7 +14,8 @@
<i class="fa fa-chevron-down"></i></a></li><div id=Integrations class=collapse><ul class=sub-menu><li><a href=../aws/>AWS</a></li><li><a href=../dell/>Dell</a></li><li><a href=../jdbc/>JDBC</a></li><li><a href=../nessie/>Nessie</a></li></ul></div><li><a class="chevron-toggle collapsed" data-toggle=collapse data-parent=full href=#API><span>API</span>
<i class="fa fa-chevron-right"></i>
<i class="fa fa-chevron-down"></i></a></li><div id=API class=collapse><ul class=sub-menu><li><a href=../java-api-quickstart/>Java Quickstart</a></li><li><a href=../api/>Java API</a></li><li><a href=../custom-catalog/>Java Custom Catalog</a></li><li><a href=../python-quickstart/>Python Quickstart</a></li><li><a href=../python-api-intro/>Python API</a></li><li><a href=../python-feature-support/>Python Feature Support</a></li></ul></div><li><a href=https://iceberg.apache.org/docs/0.14.1/../ [...]
-a <a href=https://cwiki.apache.org/confluence/display/Hive/StorageHandlers>StorageHandler</a>.</p><h2 id=feature-support>Feature support</h2><p>Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features:</p><ul><li>Creating a table</li><li>Dropping a table</li><li>Reading a table</li><li>Inserting into a table (INSERT INTO)</li></ul><div class=warning>DML operations work only with MapReduce execution engine.</div><p>With Hive version 4.0.0-alpha-1 and above,
+a <a href=https://cwiki.apache.org/confluence/display/Hive/StorageHandlers>StorageHandler</a>.</p><h2 id=feature-support>Feature support</h2><p>Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features:</p><ul><li>Creating a table</li><li>Dropping a table</li><li>Reading a table</li><li>Inserting into a table (INSERT INTO)</li></ul><div class=warning>DML operations work only with MapReduce execution engine.</div><p>With Hive version 4.0.0-alpha-2 and above,
+the Iceberg integration when using HiveCatalog supports the following additional features:</p><ul><li>Altering a table with expiring snapshots.</li><li>Create a table like an existing table (CTLT table)</li><li>Support adding parquet compression type via Table properties <a href=https://spark.apache.org/docs/2.4.3/sql-data-sources-parquet.html#configuration>Compression types</a></li><li>Altering a table metadata location</li><li>Supporting table rollback</li><li>Honours sort orders on ex [...]
the Iceberg integration when using HiveCatalog supports the following additional features:</p><ul><li>Creating an Iceberg identity-partitioned table</li><li>Creating an Iceberg table with any partition spec, including the various transforms supported by Iceberg</li><li>Creating a table from an existing table (CTAS table)</li><li>Altering a table while keeping Iceberg and Hive schemas in sync</li><li>Altering the partition schema (updating columns)</li><li>Altering the partition schema by [...]
Hive’s classpath. These are provided by the <code>iceberg-hive-runtime</code> jar file. For example, if using the Hive shell, this
can be achieved by issuing a statement like so:</p><pre tabindex=0><code>add jar /path/to/iceberg-hive-runtime.jar;
@@ -61,12 +62,13 @@ The default is Parquet:</p><div class=highlight><pre tabindex=0 style=color:#f8f
</span></span></code></pre></div><h4 id=partitioned-tables>Partitioned tables</h4><p>You can create Iceberg partitioned tables using a command familiar to those who create non-Iceberg tables:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> x (i int) PARTITIONED <sp [...]
</span></span></code></pre></div><div class=info>The resulting table does not create partitions in HMS, but instead, converts partition data into Iceberg identity partitions.</div><p>Use the DESCRIBE command to get information about the Iceberg identity partitions:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>DESC [...]
</span></span></code></pre></div><p>The result is:</p><table><thead><tr><th>col_name</th><th>data_type</th><th>comment</th></tr></thead><tbody><tr><td>i</td><td>int</td><td></td></tr><tr><td>j</td><td>int</td><td></td></tr><tr><td></td><td>NULL</td><td>NULL</td></tr><tr><td># Partition Transform Information</td><td>NULL</td><td>NULL</td></tr><tr><td># col_name</td><td>transform_type</td><td>NULL</td></tr><tr><td>j</td><td>IDENTITY</td><td>NULL</td></tr></tbody></table><p>You can create I [...]
-(supported only in Hive 4.0.0-alpha-1):</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> x (i int, ts <span style=color:#66d9ef>timestamp</span>) PARTITIONED <span style=color:#66d9ef>BY</span> SPEC (<span style=color:#66d9ef>month</span>(ts), bucket(<span style=col [...]
+(supported only from Hive 4.0.0-alpha-1):</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> x (i int, ts <span style=color:#66d9ef>timestamp</span>) PARTITIONED <span style=color:#66d9ef>BY</span> SPEC (<span style=color:#66d9ef>month</span>(ts), bucket(<span style=c [...]
</span></span><span style=display:flex><span><span style=color:#66d9ef>DESCRIBE</span> x;
</span></span></code></pre></div><p>The result is:</p><table><thead><tr><th>col_name</th><th>data_type</th><th>comment</th></tr></thead><tbody><tr><td>i</td><td>int</td><td></td></tr><tr><td>ts</td><td>timestamp</td><td></td></tr><tr><td></td><td>NULL</td><td>NULL</td></tr><tr><td># Partition Transform Information</td><td>NULL</td><td>NULL</td></tr><tr><td># col_name</td><td>transform_type</td><td>NULL</td></tr><tr><td>ts</td><td>MONTH</td><td>NULL</td></tr><tr><td>i</td><td>BUCKET[2]</t [...]
The Iceberg table and the corresponding Hive table are created at the beginning of the query execution.
The data is inserted / committed when the query finishes. So for a transient period the table already exists but contains no data.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> target PARTITIONED <span style=color:#66d9ef>BY</span> SPEC (<span style=color:#66d9ef [...]
</span></span><span style=display:flex><span> <span style=color:#66d9ef>SELECT</span> <span style=color:#f92672>*</span> <span style=color:#66d9ef>FROM</span> <span style=color:#66d9ef>source</span>;
+</span></span></code></pre></div><h3 id=create-table-like-table>CREATE TABLE LIKE TABLE</h3><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>TABLE</span> target <span style=color:#66d9ef>LIKE</span> <span style=color:#66d9ef>source</span> STORED <span style=color:#66d9ef>BY</span> [...]
</span></span></code></pre></div><h3 id=create-external-table-overlaying-an-existing-iceberg-table>CREATE EXTERNAL TABLE overlaying an existing Iceberg table</h3><p>The <code>CREATE EXTERNAL TABLE</code> command is used to overlay a Hive table “on top of” an existing Iceberg table. Iceberg
tables are created using either a <a href=../../../javadoc/0.14.1/index.html?org/apache/iceberg/catalog/Catalog.html><code>Catalog</code></a>, or an implementation of the <a href=../../../javadoc/0.14.1/index.html?org/apache/iceberg/Tables.html><code>Tables</code></a> interface, and Hive needs to be configured accordingly to
operate on these different types of table.</p><h4 id=hive-catalog-tables>Hive catalog tables</h4><p>As described before, tables created by the <code>HiveCatalog</code> with Hive engine feature enabled are directly visible by the
@@ -119,6 +121,8 @@ i.e. if columns are specified out-of-order an error will be thrown signalling th
</span></span></code></pre></div><p>During the migration the data files are not changed, only the appropriate Iceberg metadata files are created.
After the migration, handle the table as a normal Iceberg table.</p><h3 id=truncate-table>TRUNCATE TABLE</h3><p>The following command truncates the Iceberg table:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>TRUNCATE</span> <span style=color:#66d9ef>TABLE</span> t;
</span></span></code></pre></div><p>Using a partition specification is not allowed.</p><h3 id=drop-table>DROP TABLE</h3><p>Tables can be dropped using the <code>DROP TABLE</code> command:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>DROP</span> <span style=color:#66d9ef>TABLE</span> [<span style=color:#66d9ef>IF</ [...]
+</span></span></code></pre></div><h3 id=metadata-location>METADATA LOCATION</h3><p>The metadata location (snapshot location) only can be changed if the new path contains the exact same metadata json.
+It can be done only after migrating the table to Iceberg, the two operation cannot be done in one step.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>ALTER</span> <span style=color:#66d9ef>TABLE</span> t <span style=color:#66d9ef>set</span> TBLPROPERTIES (<span style=color:#e6db74>'metadata_location'</span> [...]
</span></span></code></pre></div><h2 id=dml-commands>DML Commands</h2><h3 id=select>SELECT</h3><p>Select statements work the same on Iceberg tables in Hive. You will see the Iceberg benefits over Hive in compilation and execution:</p><ul><li><strong>No file system listings</strong> - especially important on blob stores, like S3</li><li><strong>No partition listing from</strong> the Metastore</li><li><strong>Advanced partition filtering</strong> - the partition keys are not needed in the [...]
Also currently the statistics stored in the MetaStore are used for query planning. This is something we are planning to improve in the future.</p><h3 id=insert-into>INSERT INTO</h3><p>Hive supports the standard single-table INSERT INTO operation:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>INSERT</span> <span sty [...]
</span></span><span style=display:flex><span><span style=color:#66d9ef>VALUES</span> (<span style=color:#e6db74>'a'</span>, <span style=color:#ae81ff>1</span>);
@@ -140,10 +144,14 @@ To reference a metadata table the full name of the table should be used, like:
For these views it is possible to use projections / joins / filters / etc.
The function is available with the following syntax:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>SELECT</span> <span style=color:#f92672>*</span> <span style=color:#66d9ef>FROM</span> table_a <span style=color:#66d9ef>FOR</span> SYSTEM_TIME <span style=color:#66d9ef>AS</span> <span style=color:#66d9ef>OF</span> < [...]
</span></span><span style=display:flex><span><span style=color:#66d9ef>SELECT</span> <span style=color:#f92672>*</span> <span style=color:#66d9ef>FROM</span> table_a <span style=color:#66d9ef>FOR</span> SYSTEM_VERSION <span style=color:#66d9ef>AS</span> <span style=color:#66d9ef>OF</span> <span style=color:#ae81ff>1234567</span>;
-</span></span></code></pre></div><h2 id=type-compatibility>Type compatibility</h2><p>Hive and Iceberg support different set of types. Iceberg can perform type conversion automatically, but not for all
+</span></span></code></pre></div><p>You can expire snapshots of an Iceberg table using an ALTER TABLE query from Hive. You should periodically expire snapshots to delete data files that is no longer needed, and reduce the size of table metadata.</p><p>Each write to an Iceberg table from Hive creates a new snapshot, or version, of a table. Snapshots can be used for time-travel queries, or the table can be rolled back to any valid snapshot. Snapshots accumulate until they are expired by th [...]
+Enter a query to expire snapshots having the following timestamp: <code>2021-12-09 05:39:18.689000000</code></p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>ALTER</span> <span style=color:#66d9ef>TABLE</span> test_table <span style=color:#66d9ef>EXECUTE</span> expire_snapshots(<span style=color:#e6db74>'2021-12-0 [...]
+</span></span></code></pre></div><h3 id=type-compatibility>Type compatibility</h3><p>Hive and Iceberg support different set of types. Iceberg can perform type conversion automatically, but not for all
combinations, so you may want to understand the type conversion in Iceberg in prior to design the types of columns in
your tables. You can enable auto-conversion through Hadoop configuration (not enabled by default):</p><table><thead><tr><th>Config key</th><th>Default</th><th>Description</th></tr></thead><tbody><tr><td>iceberg.mr.schema.auto.conversion</td><td>false</td><td>if Hive should perform type auto-conversion</td></tr></tbody></table><h3 id=hive-type-to-iceberg-type>Hive type to Iceberg type</h3><p>This type conversion table describes how Hive types are converted to the Iceberg types. The conver [...]
-creating Iceberg table and writing to Iceberg table via Hive.</p><table><thead><tr><th>Hive</th><th>Iceberg</th><th>Notes</th></tr></thead><tbody><tr><td>boolean</td><td>boolean</td><td></td></tr><tr><td>short</td><td>integer</td><td>auto-conversion</td></tr><tr><td>byte</td><td>integer</td><td>auto-conversion</td></tr><tr><td>integer</td><td>integer</td><td></td></tr><tr><td>long</td><td>long</td><td></td></tr><tr><td>float</td><td>float</td><td></td></tr><tr><td>double</td><td>double</ [...]
+creating Iceberg table and writing to Iceberg table via Hive.</p><table><thead><tr><th>Hive</th><th>Iceberg</th><th>Notes</th></tr></thead><tbody><tr><td>boolean</td><td>boolean</td><td></td></tr><tr><td>short</td><td>integer</td><td>auto-conversion</td></tr><tr><td>byte</td><td>integer</td><td>auto-conversion</td></tr><tr><td>integer</td><td>integer</td><td></td></tr><tr><td>long</td><td>long</td><td></td></tr><tr><td>float</td><td>float</td><td></td></tr><tr><td>double</td><td>double</ [...]
+</span></span></code></pre></div><p>Rollback to a specific snapshot ID</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>ALTER</span> <span style=color:#66d9ef>TABLE</span> ice_t <span style=color:#66d9ef>EXECUTE</span> <span style=color:#66d9ef>ROLLBACK</span>(<span style=color:#ae81ff>1111</span>);
+</span></span></code></pre></div></div><div id=toc class=markdown-body><div id=full><nav id=TableOfContents><ul><li><a href=#feature-support>Feature support</a></li><li><a href=#enabling-iceberg-support-in-hive>Enabling Iceberg support in Hive</a><ul><li><a href=#hive-400-alpha-1>Hive 4.0.0-alpha-1</a></li><li><a href=#hive-23x-hive-31x>Hive 2.3.x, Hive 3.1.x</a></li></ul></li><li><a href=#catalog-management>Catalog Management</a><ul><li><a href=#global-hive-catalog>Global Hive catalog</ [...]
<script src=https://iceberg.apache.org/docs/0.14.1//js/jquery.easing.min.js></script>
<script type=text/javascript src=https://iceberg.apache.org/docs/0.14.1//js/search.js></script>
<script src=https://iceberg.apache.org/docs/0.14.1//js/bootstrap.min.js></script>
diff --git a/docs/0.14.1/index.html b/docs/0.14.1/index.html
index 35af3e09..02bf503f 100644
--- a/docs/0.14.1/index.html
+++ b/docs/0.14.1/index.html
@@ -1,4 +1,4 @@
-<!doctype html><html><head><meta name=generator content="Hugo 0.102.3"><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content><meta name=author content><title>Introduction</title><link href=./css/bootstrap.css rel=stylesheet><link href=./css/markdown.css rel=stylesheet><link href=./css/katex.min.css rel=stylesheet><link href=./css/iceberg-theme.css rel=stylesheet><link href=./f [...]
+<!doctype html><html><head><meta name=generator content="Hugo 0.108.0"><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content><meta name=author content><title>Introduction</title><link href=./css/bootstrap.css rel=stylesheet><link href=./css/markdown.css rel=stylesheet><link href=./css/katex.min.css rel=stylesheet><link href=./css/iceberg-theme.css rel=stylesheet><link href=./f [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span>
<span class=icon-bar></span>
diff --git a/docs/0.14.1/index.xml b/docs/0.14.1/index.xml
index f432bc24..9cb216e7 100644
--- a/docs/0.14.1/index.xml
+++ b/docs/0.14.1/index.xml
@@ -3,8 +3,8 @@ Spark is currently the most feature-rich compute engine for Iceberg operations.
Using Iceberg in Spark 3 To use Iceberg in a Spark shell, use the --packages option:
spark-shell --packages org.</description></item><item><title>Hive</title><link>https://iceberg.apache.org/docs/0.14.1/hive/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/0.14.1/hive/</guid><description>Hive Iceberg supports reading and writing Iceberg tables through Hive by using a StorageHandler.
Feature support Iceberg compatibility with Hive 2.x and Hive 3.1.2/3 supports the following features:
-Creating a table Dropping a table Reading a table Inserting into a table (INSERT INTO) DML operations work only with MapReduce execution engine. With Hive version 4.0.0-alpha-1 and above, the Iceberg integration when using HiveCatalog supports the following additional features:
-Creating an Iceberg identity-partitioned table Creating an Iceberg table with any partition spec, including the various transforms supported by Iceberg Creating a table from an existing table (CTAS table) Altering a table while keeping Iceberg and Hive schemas in sync Altering the partition schema (updating columns) Altering the partition schema by specifying partition transforms Truncating a table Migrating tables in Avro, Parquet, or ORC (Non-ACID) format to Iceberg Reading the schema [...]
+Creating a table Dropping a table Reading a table Inserting into a table (INSERT INTO) DML operations work only with MapReduce execution engine. With Hive version 4.0.0-alpha-2 and above, the Iceberg integration when using HiveCatalog supports the following additional features:
+Altering a table with expiring snapshots.</description></item><item><title>AWS</title><link>https://iceberg.apache.org/docs/0.14.1/aws/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://iceberg.apache.org/docs/0.14.1/aws/</guid><description>Iceberg AWS Integrations Iceberg provides integration with different AWS services through the iceberg-aws module. This section describes how to use Iceberg with AWS.
Enabling AWS Integration The iceberg-aws module is bundled with Spark and Flink engine runtimes for all versions from 0.11.0 onwards. However, the AWS clients are not bundled so that you can use the same client version as your application. You will need to provide the AWS v2 SDK because that is what Iceberg depends on.</description></item><item><title>Configuration</title><link>https://iceberg.apache.org/docs/0.14.1/configuration/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>< [...]
Read properties Property Default Description read.split.target-size 134217728 (128 MB) Target size when combining data input splits read.split.metadata-target-size 33554432 (32 MB) Target size when combining metadata input splits read.split.planning-lookback 10 Number of bins to consider when combining input splits read.split.open-file-cost 4194304 (4 MB) The estimated cost to open a file, used as a minimum weight when combining splits.</description></item><item><title>DDL</title><link>h [...]
Iceberg uses Apache Spark&rsquo;s DataSourceV2 API for data source and catalog implementations. Spark DSv2 is an evolving API with different levels of support in Spark versions. Spark 2.4 does not support SQL DDL.
diff --git a/docs/0.14.1/java-api-quickstart/index.html b/docs/0.14.1/java-api-quickstart/index.html
index f8f829a1..da97138a 100644
--- a/docs/0.14.1/java-api-quickstart/index.html
+++ b/docs/0.14.1/java-api-quickstart/index.html
@@ -61,10 +61,10 @@ You can initialize a Hive catalog with a name and some properties.
</span></span><span style=display:flex><span><span style=color:#f92672>import</span> org.apache.iceberg.types.Types<span style=color:#f92672>;</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>Schema schema <span style=color:#f92672>=</span> <span style=color:#66d9ef>new</span> Schema<span style=color:#f92672>(</span>
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>1<span style=color:#f92672>,</span> <span style=color:#e6db74>"level"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style=co [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>2<span style=color:#f92672>,</span> <span style=color:#e6db74>"event_time"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>TimestampType</span><span style=color:#f92672>.</span><span [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span>3<span style=color:#f92672>,</span> <span style=color:#e6db74>"message"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=color:#f92672>.</span><span style= [...]
-</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span>4<span style=color:#f92672>,</span> <span style=color:#e6db74>"call_stack"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>ListType</span><span style=color:#f92672>.</span><span style [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>1</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"level"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=col [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>2</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"event_time"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>TimestampType</span><span s [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>required</span><span style=color:#f92672>(</span><span style=color:#ae81ff>3</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"message"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>StringType</span><span style=c [...]
+</span></span><span style=display:flex><span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>NestedField</span><span style=color:#f92672>.</span><span style=color:#a6e22e>optional</span><span style=color:#f92672>(</span><span style=color:#ae81ff>4</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"call_stack"</span><span style=color:#f92672>,</span> Types<span style=color:#f92672>.</span><span style=color:#a6e22e>ListType</span><span style= [...]
</span></span><span style=display:flex><span> <span style=color:#f92672>);</span>
</span></span></code></pre></div><p>When using the Iceberg API directly, type IDs are required. Conversions from other schema formats, like Spark, Avro, and Parquet will automatically assign new IDs.</p><p>When a table is created, all IDs in the schema are re-assigned to ensure uniqueness.</p><h3 id=convert-a-schema-from-avro>Convert a schema from Avro</h3><p>To create an Iceberg schema from an existing Avro schema, use converters in <code>AvroSchemaUtil</code>:</p><div class=highlight>< [...]
</span></span><span style=display:flex><span><span style=color:#f92672>import</span> org.apache.avro.Schema.Parser<span style=color:#f92672>;</span>
diff --git a/docs/0.14.1/maintenance/index.html b/docs/0.14.1/maintenance/index.html
index cc2040d8..3fe5b85a 100644
--- a/docs/0.14.1/maintenance/index.html
+++ b/docs/0.14.1/maintenance/index.html
@@ -14,7 +14,7 @@
<i class="fa fa-chevron-down"></i></a></li><div id=Integrations class=collapse><ul class=sub-menu><li><a href=../aws/>AWS</a></li><li><a href=../dell/>Dell</a></li><li><a href=../jdbc/>JDBC</a></li><li><a href=../nessie/>Nessie</a></li></ul></div><li><a class="chevron-toggle collapsed" data-toggle=collapse data-parent=full href=#API><span>API</span>
<i class="fa fa-chevron-right"></i>
<i class="fa fa-chevron-down"></i></a></li><div id=API class=collapse><ul class=sub-menu><li><a href=../java-api-quickstart/>Java Quickstart</a></li><li><a href=../api/>Java API</a></li><li><a href=../custom-catalog/>Java Custom Catalog</a></li><li><a href=../python-quickstart/>Python Quickstart</a></li><li><a href=../python-api-intro/>Python API</a></li><li><a href=../python-feature-support/>Python Feature Support</a></li></ul></div><li><a href=https://iceberg.apache.org/docs/0.14.1/../ [...]
-</span></span><span style=display:flex><span><span style=color:#66d9ef>long</span> tsToExpire <span style=color:#f92672>=</span> System<span style=color:#f92672>.</span><span style=color:#a6e22e>currentTimeMillis</span><span style=color:#f92672>()</span> <span style=color:#f92672>-</span> <span style=color:#f92672>(</span>1000 <span style=color:#f92672>*</span> 60 <span style=color:#f92672>*</span> 60 <span style=color:#f92672>*</span> 24<span style=color:#f92672>);</span> <span style=co [...]
+</span></span><span style=display:flex><span><span style=color:#66d9ef>long</span> tsToExpire <span style=color:#f92672>=</span> System<span style=color:#f92672>.</span><span style=color:#a6e22e>currentTimeMillis</span><span style=color:#f92672>()</span> <span style=color:#f92672>-</span> <span style=color:#f92672>(</span><span style=color:#ae81ff>1000</span> <span style=color:#f92672>*</span> <span style=color:#ae81ff>60</span> <span style=color:#f92672>*</span> <span style=color:#ae81f [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>table<span style=color:#f92672>.</span><span style=color:#a6e22e>expireSnapshots</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>expireOlderThan</span><span style=color:#f92672>(</span>tsToExpire<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>commit</span><span style=color:#f92672>();</span>
@@ -40,13 +40,13 @@ FileSystem API to avoid unintentional deletion.</div><h2 id=optional-maintenance
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteDataFiles</span><span style=color:#f92672>(</span>table<span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>filter</span><span style=color:#f92672>(</span>Expressions<span style=color:#f92672>.</span><span style=color:#a6e22e>equal</span><span style=color:#f92672>(</span><span style=color:#e6db74>"date"</span><span style=color:#f92672>,</span> <span style=color:#e6db74>"2020-08-18"</span><span style=color:#f92672>))</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>option</span><span style=color:#f92672>(</span><span style=color:#e6db74>"target-file-size-bytes"</span><span style=color:#f92672>,</span> Long<span style=color:#f92672>.</span><span style=color:#a6e22e>toString</span><span style=color:#f92672>(</span>500 <span style=color:#f92672>*</span> 1024 <span style=color:#f92672>*</span> 1024<span style=color:#f92672>))</span> <spa [...]
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>option</span><span style=color:#f92672>(</span><span style=color:#e6db74>"target-file-size-bytes"</span><span style=color:#f92672>,</span> Long<span style=color:#f92672>.</span><span style=color:#a6e22e>toString</span><span style=color:#f92672>(</span><span style=color:#ae81ff>500</span> <span style=color:#f92672>*</span> <span style=color:#ae81ff>1024</span> <span style=c [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> <span style=color:#f92672>.</span><span style=color:#a6e22e>execute</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>The <code>files</code> metadata table is useful for inspecting data file sizes and determining when to compact partitions.</p><p>See the <a href=../../../javadoc/0.14.1/org/apache/iceberg/actions/RewriteDataFiles.html><code>RewriteDataFiles</code> Javadoc</a> to see more configuration options.</p><h3 id=rewrite-manifests>Rewrite manifests</h3><p>Iceberg uses metadata in its manifest list and manifest files speed up query planning and to prune unnecessa [...]
</span></span><span style=display:flex><span>SparkActions
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>get</span><span style=color:#f92672>()</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteManifests</span><span style=color:#f92672>(</span>table<span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteIf</span><span style=color:#f92672>(</span>file <span style=color:#f92672>-></span> file<span style=color:#f92672>.</span><span style=color:#a6e22e>length</span><span style=color:#f92672>()</span> <span style=color:#f92672><</span> 10 <span style=color:#f92672>*</span> 1024 <span style=color:#f92672>*</span> 1024<span style=color:#f92672>)</span> <span style=color:#75 [...]
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>rewriteIf</span><span style=color:#f92672>(</span>file <span style=color:#f92672>-></span> file<span style=color:#f92672>.</span><span style=color:#a6e22e>length</span><span style=color:#f92672>()</span> <span style=color:#f92672><</span> <span style=color:#ae81ff>10</span> <span style=color:#f92672>*</span> <span style=color:#ae81ff>1024</span> <span style=color:#f92672>*</ [...]
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> <span style=color:#f92672>.</span><span style=color:#a6e22e>execute</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>See the <a href=../../../javadoc/0.14.1/org/apache/iceberg/actions/RewriteManifests.html><code>RewriteManifests</code> Javadoc</a> to see more configuration options.</p></div><div id=toc class=markdown-body><div id=full><nav id=TableOfContents><ul><li><a href=#recommended-maintenance>Recommended Maintenance</a><ul><li><a href=#expire-snapshots>Expire Snapshots</a></li><li><a href=#remove-old-metadata-files>Remove old metadata files</a></li><li><a href= [...]
<script src=https://iceberg.apache.org/docs/0.14.1//js/jquery.easing.min.js></script>