You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by bl...@apache.org on 2021/01/26 23:14:56 UTC
[iceberg] branch asf-site updated: Deployed e5e1c8a6e with MkDocs
version: 1.0.4
This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d828cca Deployed e5e1c8a6e with MkDocs version: 1.0.4
d828cca is described below
commit d828ccac011a14e9d4625b4050194a3fee46dd67
Author: Ryan Blue <bl...@apache.org>
AuthorDate: Tue Jan 26 15:14:24 2021 -0800
Deployed e5e1c8a6e with MkDocs version: 1.0.4
---
404.html | 31 +-
api/index.html | 51 ++-
aws/index.html | 744 ++++++++++++++++++++++++++++++++++
community/index.html | 31 +-
configuration/index.html | 133 +++++-
css/extra.css | 4 +
custom-catalog/index.html | 31 +-
evolution/index.html | 61 ++-
flink/index.html | 149 ++++++-
getting-started/index.html | 31 +-
hive/index.html | 71 +++-
how-to-release/index.html | 31 +-
index.html | 42 +-
java-api-quickstart/index.html | 33 +-
maintenance/index.html | 31 +-
partitioning/index.html | 35 +-
performance/index.html | 31 +-
presto/index.html | 434 +-------------------
python-api-intro/index.html | 31 +-
python-feature-support/index.html | 31 +-
python-quickstart/index.html | 31 +-
releases/index.html | 31 +-
reliability/index.html | 31 +-
schemas/index.html | 31 +-
sitemap.xml | 55 +--
sitemap.xml.gz | Bin 226 -> 230 bytes
snapshots/index.html | 31 +-
spark-structured-streaming/index.html | 51 ++-
spark/index.html | 687 ++++++++++++++++++++++++++++++-
spec/index.html | 158 +++++++-
terms/index.html | 31 +-
trademarks/index.html | 31 +-
{snapshots => trino}/index.html | 44 +-
why-iceberg/index.html | 31 +-
34 files changed, 2540 insertions(+), 739 deletions(-)
diff --git a/404.html b/404.html
index f4fd975..97f8e78 100644
--- a/404.html
+++ b/404.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="/.">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="/.">About</a>
+</li>
+
+
+
+<li >
<a href="/community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="/aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -354,7 +369,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/api/index.html b/api/index.html
index 1bea4fb..0ef792e 100644
--- a/api/index.html
+++ b/api/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -212,6 +212,21 @@
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -332,7 +347,7 @@
<ul class="nav navbar-nav navbar-right">
<li >
- <a rel="prev" href="../hive/">
+ <a rel="prev" href="../aws/">
<i class="fas fa-arrow-left"></i> Previous
</a>
</li>
@@ -392,9 +407,9 @@
<h1 id="iceberg-java-api">Iceberg Java API<a class="headerlink" href="#iceberg-java-api" title="Permanent link">¶</a></h1>
<h2 id="tables">Tables<a class="headerlink" href="#tables" title="Permanent link">¶</a></h2>
<p>The main purpose of the Iceberg API is to manage table metadata, like schema, partition spec, metadata, and data files that store table data.</p>
-<p>Table metadata and operations are accessed through the <code>Table</code> interface. This interface will return table information, </p>
+<p>Table metadata and operations are accessed through the <code>Table</code> interface. This interface will return table information.</p>
<h3 id="table-metadata">Table metadata<a class="headerlink" href="#table-metadata" title="Permanent link">¶</a></h3>
-<p>The <a href="../javadoc/master/?org/apache/iceberg/Table.html"><code>Table</code> interface</a> provides access table metadata:</p>
+<p>The <a href="../javadoc/master/?org/apache/iceberg/Table.html"><code>Table</code> interface</a> provides access to the table metadata:</p>
<ul>
<li><code>schema</code> returns the current table <a href="../schemas/">schema</a></li>
<li><code>spec</code> returns the current table partition spec</li>
@@ -449,12 +464,12 @@ Iterable<CombinedScanTask> tasks = scan.planTasks();
<li><code>newRewrite</code> – used to rewrite data files; will replace existing files with new versions</li>
<li><code>newTransaction</code> – create a new table-level transaction</li>
<li><code>rewriteManifests</code> – rewrite manifest data by clustering files, for faster scan planning</li>
-<li><code>rollback</code> – roll the table state back by pointing current to a specific snapshot</li>
+<li><code>rollback</code> – rollback the table state to a specific snapshot</li>
</ul>
<h3 id="transactions">Transactions<a class="headerlink" href="#transactions" title="Permanent link">¶</a></h3>
<p>Transactions are used to commit multiple table changes in a single atomic operation. A transaction is used to create individual operations using factory methods, like <code>newAppend</code>, just like working with a <code>Table</code>. Operations created by a transaction are committed as a group when <code>commitTransaction</code> is called.</p>
<p>For example, deleting and appending a file in the same transaction:</p>
-<pre><code class="java">Transaction t = table.newTrasaction();
+<pre><code class="java">Transaction t = table.newTransaction();
// commit operations to the transaction
t.newDelete().deleteFromRowFilter(filter).commit();
@@ -524,7 +539,7 @@ ListType list = ListType.ofRequired(1, IntegerType.get());
</ul>
<h3 id="expression-binding">Expression binding<a class="headerlink" href="#expression-binding" title="Permanent link">¶</a></h3>
<p>When created, expressions are unbound. Before an expression is used, it will be bound to a data type to find the field ID the expression name represents, and to convert predicate literals.</p>
-<p>For example, before using the expression <code>lessThan("x", 10)</code>, Iceberg needs to determine which column <code>"x"</code> refers to and convert <code>10</code> to that colunn’s data type.</p>
+<p>For example, before using the expression <code>lessThan("x", 10)</code>, Iceberg needs to determine which column <code>"x"</code> refers to and convert <code>10</code> to that column’s data type.</p>
<p>If the expression could be bound to the type <code>struct<1 x: long, 2 y: long></code> or to <code>struct<11 x: int, 12 y: int></code>.</p>
<h3 id="expression-example">Expression example<a class="headerlink" href="#expression-example" title="Permanent link">¶</a></h3>
<pre><code class="java">table.newScan()
@@ -537,15 +552,21 @@ ListType list = ListType.ofRequired(1, IntegerType.get());
<ul>
<li><code>iceberg-common</code> contains utility classes used in other modules</li>
<li><code>iceberg-api</code> contains the public Iceberg API, including expressions, types, tables, and operations</li>
+<li><code>iceberg-arrow</code> is an implementation of the Iceberg type system for reading and writing data stored in Iceberg tables using Apache Arrow as the in-memory data format</li>
+<li><code>iceberg-aws</code> contains implementations of the Iceberg API to be used with tables stored on AWS S3 and/or for tables defined using the AWS Glue data catalog</li>
<li><code>iceberg-core</code> contains implementations of the Iceberg API and support for Avro data files, <strong>this is what processing engines should depend on</strong></li>
<li><code>iceberg-parquet</code> is an optional module for working with tables backed by Parquet files</li>
<li><code>iceberg-orc</code> is an optional module for working with tables backed by ORC files (<em>experimental</em>)</li>
<li><code>iceberg-hive-metastore</code> is an implementation of Iceberg tables backed by the Hive metastore Thrift client</li>
</ul>
-<p>This project Iceberg also has modules for adding Iceberg support to processing engines:</p>
+<p>This project Iceberg also has modules for adding Iceberg support to processing engines and associated tooling:</p>
<ul>
<li><code>iceberg-spark2</code> is an implementation of Spark’s Datasource V2 API in 2.4 for Iceberg (use iceberg-spark-runtime for a shaded version)</li>
<li><code>iceberg-spark3</code> is an implementation of Spark’s Datasource V2 API in 3.0 for Iceberg (use iceberg-spark3-runtime for a shaded version)</li>
+<li><code>iceberg-flink</code> is an implementation of Flink’s Table and DataStream API for Iceberg (use iceberg-flink-runtime for a shaded version)</li>
+<li><code>iceberg-hive3</code> is an implementation of Hive 3 specific SerDe’s for Timestamp, TimestampWithZone, and Date object inspectors (use iceberg-hive-runtime for a shaded version).</li>
+<li><code>iceberg-mr</code> is an implementation of MapReduce and Hive InputFormats and SerDes for Iceberg (use iceberg-hive-runtime for a shaded version for use with Hive)</li>
+<li><code>iceberg-nessie</code> is a module used to integrate Iceberg table metadata history and operations with <a href="https://projectnessie.org/">Project Nessie</a></li>
<li><code>iceberg-data</code> is a client library used to read Iceberg tables from JVM applications</li>
<li><code>iceberg-pig</code> is an implementation of Pig’s LoadFunc API for Iceberg</li>
<li><code>iceberg-runtime</code> generates a shaded runtime jar for Spark to integrate with iceberg tables</li>
@@ -558,7 +579,7 @@ ListType list = ListType.ofRequired(1, IntegerType.get());
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/aws/index.html b/aws/index.html
new file mode 100644
index 0000000..2d44b3b
--- /dev/null
+++ b/aws/index.html
@@ -0,0 +1,744 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+ <meta name="description" content="A table format for large, slow-moving tabular data">
+
+
+ <link rel="../img/favicon.ico">
+
+
+ <title>AWS - Apache Iceberg</title>
+
+
+ <link rel="stylesheet" href="//use.fontawesome.com/releases/v5.5.0/css/all.css" integrity="sha384-B4dIYHKNBt8Bc12p+WXckhzcICo0wtJAoU8YZTY5qE0Id1GSseTk6S+L3BlXeVIU" crossorigin="anonymous">
+ <link rel="stylesheet" href="//cdn.jsdelivr.net/npm/hack-font@3.3.0/build/web/hack.min.css">
+ <link href='//fonts.googleapis.com/css?family=PT+Sans:400,400italic,700,700italic&subset=latin-ext,latin' rel='stylesheet' type='text/css'>
+ <link href='//fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,300,600,700&subset=latin-ext,latin' rel='stylesheet' type='text/css'>
+ <link href="../css/bootstrap-custom.min.css" rel="stylesheet">
+ <link href="../css/base.min.css" rel="stylesheet">
+ <link href="../css/cinder.min.css" rel="stylesheet">
+ <link href="../css/highlight.min.css" rel="stylesheet">
+ <link href="../css/extra.css" rel="stylesheet">
+
+ <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://cdn.jsdelivr.net/npm/html5shiv@3.7.3/dist/html5shiv.min.js"></script>
+ <script src="https://cdn.jsdelivr.net/npm/respond.js@1.4.2/dest/respond.min.js"></script>
+ <![endif]-->
+
+ <script src="//ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js"></script>
+ <script>
+ WebFont.load({
+ google: {
+ families: ['Open Sans', 'PT Sans']
+ }
+ });
+ </script>
+
+
+
+
+</head>
+
+<body>
+
+ <div class="navbar navbar-default navbar-fixed-top" role="navigation">
+ <div class="container">
+
+ <!-- Collapsed navigation -->
+ <div class="navbar-header">
+ <!-- Expander button -->
+ <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+
+ <!-- Main title -->
+
+ <a class="navbar-brand" href="..">Apache Iceberg</a>
+ </div>
+
+ <!-- Expanded navigation -->
+ <div class="navbar-collapse collapse">
+ <!-- Main navigation -->
+ <ul class="nav navbar-nav">
+
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
+ <a href="../community/">Community</a>
+</li>
+
+
+
+<li >
+ <a href="../releases/">Releases</a>
+</li>
+
+
+
+<li >
+ <a href="../trademarks/">Trademarks</a>
+</li>
+
+
+
+<li >
+ <a href="../how-to-release/">How to Release</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Tables <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../configuration/">Configuration</a>
+</li>
+
+
+
+<li >
+ <a href="../schemas/">Schemas</a>
+</li>
+
+
+
+<li >
+ <a href="../partitioning/">Partitioning</a>
+</li>
+
+
+
+<li >
+ <a href="../evolution/">Table evolution</a>
+</li>
+
+
+
+<li >
+ <a href="../maintenance/">Maintenance</a>
+</li>
+
+
+
+<li >
+ <a href="../performance/">Performance</a>
+</li>
+
+
+
+<li >
+ <a href="../reliability/">Reliability</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Spark <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../getting-started/">Getting Started</a>
+</li>
+
+
+
+<li >
+ <a href="../spark/">Spark SQL</a>
+</li>
+
+
+
+<li >
+ <a href="../spark-structured-streaming/">Spark Streaming</a>
+</li>
+
+
+
+<li >
+ <a href="../spark#time-travel">Time Travel</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li >
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
+ </li>
+
+
+
+ <li >
+ <a href="../flink/">Flink</a>
+ </li>
+
+
+
+ <li >
+ <a href="../hive/">Hive</a>
+ </li>
+
+
+
+ <li class="dropdown active">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li class="active">
+ <a href="./">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="/javadoc/">Javadoc</a>
+</li>
+
+
+
+<li >
+ <a href="../api/">Java API intro</a>
+</li>
+
+
+
+<li >
+ <a href="../java-api-quickstart/">Java Quickstart</a>
+</li>
+
+
+
+<li >
+ <a href="../custom-catalog/">Java Custom Catalog</a>
+</li>
+
+
+
+<li >
+ <a href="../python-quickstart/">Python Quickstart</a>
+</li>
+
+
+
+<li >
+ <a href="../python-api-intro/">Python API Intro</a>
+</li>
+
+
+
+<li >
+ <a href="../python-feature-support/">Python Feature Support</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Format <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../terms/">Definitions</a>
+</li>
+
+
+
+<li >
+ <a href="../spec/">Spec</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li >
+ <a href="https://github.com/apache/iceberg">GitHub</a>
+ </li>
+
+
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="https://www.apache.org/licenses/">License</a>
+</li>
+
+
+
+<li >
+ <a href="https://www.apache.org/security/">Security</a>
+</li>
+
+
+
+<li >
+ <a href="https://www.apache.org/foundation/thanks.html">Sponsors</a>
+</li>
+
+
+
+<li >
+ <a href="https://www.apache.org/foundation/sponsorship.html">Donate</a>
+</li>
+
+
+
+<li >
+ <a href="https://www.apache.org/events/current-event.html">Events</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+ </ul>
+
+ <ul class="nav navbar-nav navbar-right">
+ <li >
+ <a rel="prev" href="../hive/">
+ <i class="fas fa-arrow-left"></i> Previous
+ </a>
+ </li>
+ <li >
+ <a rel="next" href="../api/">
+ Next <i class="fas fa-arrow-right"></i>
+ </a>
+ </li>
+ </ul>
+ </div>
+ </div>
+</div>
+
+ <div class="container">
+
+
+ <div class="col-md-3"><div class="bs-sidebar hidden-print affix well" role="complementary">
+ <ul class="nav bs-sidenav">
+ <li class="first-level active"><a href="#iceberg-aws-integrations">Iceberg AWS Integrations</a></li>
+ <li class="second-level"><a href="#enabling-aws-integration">Enabling AWS Integration</a></li>
+
+ <li class="second-level"><a href="#glue-catalog">Glue Catalog</a></li>
+
+ <li class="third-level"><a href="#glue-catalog-id">Glue Catalog ID</a></li>
+ <li class="third-level"><a href="#skip-archive">Skip Archive</a></li>
+ <li class="third-level"><a href="#dynamodb-for-commit-locking">DynamoDB for Commit Locking</a></li>
+ <li class="third-level"><a href="#warehouse-location">Warehouse Location</a></li>
+ <li class="third-level"><a href="#table-location">Table Location</a></li>
+ <li class="second-level"><a href="#s3-fileio">S3 FileIO</a></li>
+
+ <li class="third-level"><a href="#progressive-multipart-upload">Progressive Multipart Upload</a></li>
+ <li class="third-level"><a href="#s3-server-side-encryption">S3 Server Side Encryption</a></li>
+ <li class="third-level"><a href="#s3-access-control-list">S3 Access Control List</a></li>
+ <li class="third-level"><a href="#object-store-file-layout">Object Store File Layout</a></li>
+ <li class="third-level"><a href="#s3-strong-consistency">S3 Strong Consistency</a></li>
+ <li class="third-level"><a href="#hadoop-s3a-filesystem">Hadoop S3A FileSystem</a></li>
+ <li class="second-level"><a href="#aws-client-customization">AWS Client Customization</a></li>
+
+ <li class="third-level"><a href="#cross-account-and-cross-region-access">Cross-Account and Cross-Region Access</a></li>
+ <li class="second-level"><a href="#run-iceberg-on-aws">Run Iceberg on AWS</a></li>
+
+ </ul>
+</div></div>
+ <div class="col-md-9" role="main">
+
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+<h1 id="iceberg-aws-integrations">Iceberg AWS Integrations<a class="headerlink" href="#iceberg-aws-integrations" title="Permanent link">¶</a></h1>
+<p>Iceberg provides integration with different AWS services through the <code>iceberg-aws</code> module.
+This section describes how to use Iceberg with AWS.</p>
+<h2 id="enabling-aws-integration">Enabling AWS Integration<a class="headerlink" href="#enabling-aws-integration" title="Permanent link">¶</a></h2>
+<p>The <code>iceberg-aws</code> module is bundled with Spark and Flink engine runtimes for all versions from <code>0.11.0</code> onwards.
+However, the AWS clients are not bundled so that you can use the same client version as your application.
+You will need to provide the AWS v2 SDK because that is what Iceberg depends on.
+You can choose to use the <a href="https://mvnrepository.com/artifact/software.amazon.awssdk/bundle">AWS SDK bundle</a>,
+or individual AWS client packages (Glue, S3, DynamoDB, KMS, STS) if you would like to have a minimal dependency footprint.</p>
+<p>All the default AWS clients use the <a href="https://mvnrepository.com/artifact/software.amazon.awssdk/url-connection-client">URL Connection HTTP Client</a>
+for HTTP connection management.
+This dependency is not part of the AWS SDK bundle and needs to be added separately.
+To choose a different HTTP client library such as <a href="https://mvnrepository.com/artifact/software.amazon.awssdk/apache-client">Apache HTTP Client</a>,
+see the section <a href="#aws-client-customization">client customization</a> for more details.</p>
+<p>For example, to use AWS features with Spark 3 and AWS clients version 2.15.40, you can start the Spark SQL shell with:</p>
+<pre><code class="sh">DEPENDENCIES="org.apache.iceberg:iceberg-spark3-runtime:0.11.0"
+DEPENDENCIES+=",software.amazon.awssdk:bundle:2.15.40"
+DEPENDENCIES+=",software.amazon.awssdk:url-connection-client:2.15.40"
+
+spark-sql --packages $DEPENDENCIES \
+ --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
+ --conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
+ --conf spark.sql.catalog.my_catalog.lock-impl=org.apache.iceberg.aws.glue.DynamoLockManager \
+ --conf spark.sql.catalog.my_catalog.lock.table=myGlueLockTable
+</code></pre>
+
+<p>As you can see, In the shell command, we use <code>--packages</code> to specify the additional AWS bundle and HTTP client dependencies with their version as <code>2.15.40</code>.</p>
+<h2 id="glue-catalog">Glue Catalog<a class="headerlink" href="#glue-catalog" title="Permanent link">¶</a></h2>
+<p>Iceberg enables the use of <a href="https://aws.amazon.com/glue">AWS Glue</a> as the <code>Catalog</code> implementation.
+When used, an Iceberg namespace is stored as a <a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-databases.html">Glue Database</a>,
+an Iceberg table is stored as a <a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html">Glue Table</a>,
+and every Iceberg table version is stored as a <a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html#aws-glue-api-catalog-tables-TableVersion">Glue TableVersion</a>.
+You can start using Glue catalog by specifying the <code>catalog-impl</code> as <code>org.apache.iceberg.aws.glue.GlueCatalog</code>,
+just like what is shown in the <a href="#enabling-aws-integration">enabling AWS integration</a> section above.
+More details about loading the catalog can be found in individual engine pages, such as <a href="../spark/#loading-a-custom-catalog">Spark</a> and <a href="../flink/#creating-catalogs-and-using-catalogs">Flink</a>.</p>
+<h3 id="glue-catalog-id">Glue Catalog ID<a class="headerlink" href="#glue-catalog-id" title="Permanent link">¶</a></h3>
+<p>There is a unique Glue metastore in each AWS account and each AWS region.
+By default, <code>GlueCatalog</code> chooses the Glue metastore to use based on the user’s default AWS client credential and region setup.
+You can specify the Glue catalog ID through <code>glue.id</code> catalog property to point to a Glue catalog in a different AWS account.
+The Glue catalog ID is your numeric AWS account ID.
+If the Glue catalog is in a different region, you should configure you AWS client to point to the correct region,
+see more details in <a href="#aws-client-customization">AWS client customization</a>.</p>
+<h3 id="skip-archive">Skip Archive<a class="headerlink" href="#skip-archive" title="Permanent link">¶</a></h3>
+<p>By default, Glue stores all the table versions created and user can rollback a table to any historical version if needed.
+However, if you are streaming data to Iceberg, this will easily create a lot of Glue table versions.
+Therefore, it is recommended to turn off the archive feature in Glue by setting <code>glue.skip-archive</code> to <code>true</code>.
+For more details, please read <a href="https://docs.aws.amazon.com/general/latest/gr/glue.html">Glue Quotas</a> and the <a href="https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateTable.html">UpdateTable API</a>.</p>
+<h3 id="dynamodb-for-commit-locking">DynamoDB for Commit Locking<a class="headerlink" href="#dynamodb-for-commit-locking" title="Permanent link">¶</a></h3>
+<p>Glue does not have a strong guarantee over concurrent updates to a table.
+Although it throws <code>ConcurrentModificationException</code> when detecting two processes updating a table at the same time,
+there is no guarantee that one update would not clobber the other update.
+Therefore, <a href="https://aws.amazon.com/dynamodb">DynamoDB</a> can be used for Glue, so that for every commit,
+<code>GlueCatalog</code> first obtains a lock using a helper DynamoDB table and then try to safely modify the Glue table.</p>
+<p>This feature requires the following lock related catalog properties:</p>
+<ol>
+<li>Set <code>lock-impl</code> as <code>org.apache.iceberg.aws.glue.DynamoLockManager</code>.</li>
+<li>Set <code>lock.table</code> as the DynamoDB table name you would like to use. If the lock table with the given name does not exist in DynamoDB, a new table is created with billing mode set as <a href="https://aws.amazon.com/blogs/aws/amazon-dynamodb-on-demand-no-capacity-planning-and-pay-per-request-pricing">pay-per-request</a>.</li>
+</ol>
+<p>Other lock related catalog properties can also be used to adjust locking behaviors such as heartbeat interval.
+For more details, please refer to <a href="../configuration/#lock-catalog-properties">Lock catalog properties</a>. </p>
+<h3 id="warehouse-location">Warehouse Location<a class="headerlink" href="#warehouse-location" title="Permanent link">¶</a></h3>
+<p>Similar to all other catalog implementations, <code>warehouse</code> is a required catalog property to determine the root path of the data warehouse in storage.
+By default, Glue only allows a warehouse location in S3 because of the use of <code>S3FileIO</code>.
+To store data in a different local or cloud store, Glue catalog can switch to use <code>HadoopFileIO</code> or any custom FileIO by setting the <code>io-impl</code> catalog property.
+Details about this feature can be found in the <a href="../custom-catalog/#custom-file-io-implementation">custom FileIO</a> section.</p>
+<h3 id="table-location">Table Location<a class="headerlink" href="#table-location" title="Permanent link">¶</a></h3>
+<p>By default, the root location for a table <code>my_table</code> of namespace <code>my_ns</code> is at <code>my-warehouse-location/my-ns.db/my-table</code>.
+This default root location can be changed at both namespace and table level.</p>
+<p>To use a different path prefix for all tables under a namespace, use AWS console or any AWS Glue client SDK you like to update the <code>locationUri</code> attribute of the corresponding Glue database.
+For example, you can update the <code>locationUri</code> of <code>my_ns</code> to <code>s3://my-ns-bucket</code>,
+then any newly created table will have a default root location under the new prefix.
+For instance, a new table <code>my_table_2</code> will have its root location at <code>s3://my-ns-bucket/my_table_2</code>.</p>
+<p>To use a completely different root path for a specific table, set the <code>location</code> table property to the desired root path value you want.
+For example, in Spark SQL you can do:</p>
+<pre><code class="sql">CREATE TABLE my_catalog.my_ns.my_table (
+ id bigint,
+ data string,
+ category string)
+USING iceberg
+OPTIONS ('location'='s3://my-special-table-bucket')
+PARTITIONED BY (category);
+</code></pre>
+
+<p>For engines like Spark that supports the <code>LOCATION</code> keyword, the above SQL statement is equivalent to:</p>
+<pre><code class="sql">CREATE TABLE my_catalog.my_ns.my_table (
+ id bigint,
+ data string,
+ category string)
+USING iceberg
+LOCATION 's3://my-special-table-bucket'
+PARTITIONED BY (category);
+</code></pre>
+
+<h2 id="s3-fileio">S3 FileIO<a class="headerlink" href="#s3-fileio" title="Permanent link">¶</a></h2>
+<p>Iceberg allows users to write data to S3 through <code>S3FileIO</code>.
+<code>GlueCatalog</code> by default uses this <code>FileIO</code>, and other catalogs can load this <code>FileIO</code> using the <code>io-impl</code> catalog property.</p>
+<h3 id="progressive-multipart-upload">Progressive Multipart Upload<a class="headerlink" href="#progressive-multipart-upload" title="Permanent link">¶</a></h3>
+<p><code>S3FileIO</code> implements a customized progressive multipart upload algorithm to upload data.
+Data files are uploaded by parts in parallel as soon as each part is ready,
+and each file part is deleted as soon as its upload process completes.
+This provides maximized upload speed and minimized local disk usage during uploads.
+Here are the configurations that users can tune related to this feature:</p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Default</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>s3.multipart.num-threads</td>
+<td>the available number of processors in the system</td>
+<td>number of threads to use for uploading parts to S3 (shared across all output streams)</td>
+</tr>
+<tr>
+<td>s3.multipart.part-size-bytes</td>
+<td>32MB</td>
+<td>the size of a single part for multipart upload requests</td>
+</tr>
+<tr>
+<td>s3.multipart.threshold</td>
+<td>1.5</td>
+<td>the threshold expressed as a factor times the multipart size at which to switch from uploading using a single put object request to uploading using multipart upload</td>
+</tr>
+<tr>
+<td>s3.staging-dir</td>
+<td><code>java.io.tmpdir</code> property value</td>
+<td>the directory to hold temporary files</td>
+</tr>
+</tbody>
+</table>
+<h3 id="s3-server-side-encryption">S3 Server Side Encryption<a class="headerlink" href="#s3-server-side-encryption" title="Permanent link">¶</a></h3>
+<p><code>S3FileIO</code> supports all 3 S3 server side encryption modes:</p>
+<ul>
+<li><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html">SSE-S3</a>: When you use Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3), each object is encrypted with a unique key. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 server-side encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.</li>
+<li><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html">SSE-KMS</a>: Server-Side Encryption with Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS) is similar to SSE-S3, but with some additional benefits and charges for using this service. There are separate permissions for the use of a CMK that provides added protection against unauthorized access of your objects in Amazon S3. SSE-KMS also provides you with an audit trail that shows [...]
+<li><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html">SSE-C</a>: With Server-Side Encryption with Customer-Provided Keys (SSE-C), you manage the encryption keys and Amazon S3 manages the encryption, as it writes to disks, and decryption, when you access your objects.</li>
+</ul>
+<p>To enable server side encryption, use the following configuration properties:</p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Default</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>s3.sse.type</td>
+<td><code>none</code></td>
+<td><code>none</code>, <code>s3</code>, <code>kms</code> or <code>custom</code></td>
+</tr>
+<tr>
+<td>s3.sse.key</td>
+<td><code>aws/s3</code> for <code>kms</code> type, null otherwise</td>
+<td>A KMS Key ID or ARN for <code>kms</code> type, or a custom base-64 AES256 symmetric key for <code>custom</code> type.</td>
+</tr>
+<tr>
+<td>s3.sse.md5</td>
+<td>null</td>
+<td>If SSE type is <code>custom</code>, this value must be set as the base-64 MD5 digest of the symmetric key to ensure integrity.</td>
+</tr>
+</tbody>
+</table>
+<h3 id="s3-access-control-list">S3 Access Control List<a class="headerlink" href="#s3-access-control-list" title="Permanent link">¶</a></h3>
+<p><code>S3FileIO</code> supports S3 access control list (ACL) for detailed access control.
+User can choose the ACL level by setting the <code>s3.acl</code> property.
+For more details, please read <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html">S3 ACL Documentation</a>.</p>
+<h3 id="object-store-file-layout">Object Store File Layout<a class="headerlink" href="#object-store-file-layout" title="Permanent link">¶</a></h3>
+<p>S3 and many other cloud storage services <a href="https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/">throttle requests based on object prefix</a>.
+This means data stored in a traditional Hive storage layout has bad read and write throughput since data files of the same partition are placed under the same prefix.
+Iceberg by default uses the Hive storage layout, but can be switched to use a different <code>ObjectStoreLocationProvider</code>.
+In this mode, a hash string is added to the beginning of each file path, so that files are equally distributed across all prefixes in an S3 bucket.
+This results in minimized throttling and maximized throughput for S3-related IO operations.
+Here is an example Spark SQL command to create a table with this feature enabled:</p>
+<pre><code class="sql">CREATE TABLE my_catalog.my_ns.my_table (
+ id bigint,
+ data string,
+ category string)
+USING iceberg
+OPTIONS (
+ 'write.object-storage.enabled'=true,
+ 'write.object-storage.path'='s3://my-table-data-bucket')
+PARTITIONED BY (category);
+</code></pre>
+
+<p>For more details, please refer to the <a href="../custom-catalog/#custom-location-provider-implementation">LocationProvider Configuration</a> section. </p>
+<h3 id="s3-strong-consistency">S3 Strong Consistency<a class="headerlink" href="#s3-strong-consistency" title="Permanent link">¶</a></h3>
+<p>In November 2020, S3 announced <a href="https://aws.amazon.com/s3/consistency/">strong consistency</a> for all read operations, and Iceberg is updated to fully leverage this feature.
+There is no redundant consistency wait and check which might negatively impact performance during IO operations.</p>
+<h3 id="hadoop-s3a-filesystem">Hadoop S3A FileSystem<a class="headerlink" href="#hadoop-s3a-filesystem" title="Permanent link">¶</a></h3>
+<p>Before <code>S3FileIO</code> was introduced, many Iceberg users choose to use <code>HadoopFileIO</code> to write data to S3 through the <a href="https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java">S3A FileSystem</a>.
+As introduced in the previous sections, <code>S3FileIO</code> adopts latest AWS clients and S3 features for optimized security and performance,
+ and is thus recommend for S3 use cases rather than the S3A FileSystem.</p>
+<p><code>S3FileIO</code> writes data with <code>s3://</code> URI scheme, but it is also compatible with schemes written by the S3A FileSystem.
+This means for any table manifests containing <code>s3a://</code> or <code>s3n://</code> file paths, <code>S3FileIO</code> is still able to read them.
+This feature allows people to easily switch from S3A to <code>S3FileIO</code>.</p>
+<p>If for any reason you have to use S3A, here are the instructions:</p>
+<ol>
+<li>To store data using S3A, specify the <code>warehouse</code> catalog property to be an S3A path, e.g. <code>s3a://my-bucket/my-warehouse</code> </li>
+<li>For <code>HiveCatalog</code>, to also store metadata using S3A, specify the Hadoop config property <code>hive.metastore.warehouse.dir</code> to be an S3A path.</li>
+<li>Add <a href="https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws">hadoop-aws</a> as a runtime dependency of your compute engine.</li>
+<li>Configure AWS settings based on <a href="https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html">hadoop-aws documentation</a> (make sure you check the version, S3A configuration varies a lot based on the version you use). </li>
+</ol>
+<h2 id="aws-client-customization">AWS Client Customization<a class="headerlink" href="#aws-client-customization" title="Permanent link">¶</a></h2>
+<p>Many organizations have customized their way of configuring AWS clients with their own credential provider, access proxy, retry strategy, etc.
+Iceberg allows users to plug in their own implementation of <code>org.apache.iceberg.aws.AwsClientFactory</code> by setting the <code>client.factory</code> catalog property.</p>
+<h3 id="cross-account-and-cross-region-access">Cross-Account and Cross-Region Access<a class="headerlink" href="#cross-account-and-cross-region-access" title="Permanent link">¶</a></h3>
+<p>It is a common use case for organizations to have a centralized AWS account for Glue metastore and S3 buckets, and use different AWS accounts and regions for different teams to access those resources.
+In this case, a <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html">cross-account IAM role</a> is needed to access those centralized resources.
+Iceberg provides an AWS client factory <code>AssumeRoleAwsClientFactory</code> to support this common use case.
+This also serves as an example for users who would like to implement their own AWS client factory.</p>
+<p>This client factory has the following configurable catalog properties:</p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Default</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>client.assume-role.arn</td>
+<td>null, requires user input</td>
+<td>ARN of the role to assume, e.g. arn:aws:iam::123456789:role/myRoleToAssume</td>
+</tr>
+<tr>
+<td>client.assume-role.region</td>
+<td>null, requires user input</td>
+<td>All AWS clients except the STS client will use the given region instead of the default region chain</td>
+</tr>
+<tr>
+<td>client.assume-role.external-id</td>
+<td>null</td>
+<td>An optional <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html">external ID</a></td>
+</tr>
+<tr>
+<td>client.assume-role.timeout-sec</td>
+<td>1 hour</td>
+<td>Timeout of each assume role session. At the end of the timeout, a new set of role session credentials will be fetched through a STS client.</td>
+</tr>
+</tbody>
+</table>
+<p>By using this client factory, an STS client is initialized with the default credential and region to assume the specified role.
+The Glue, S3 and DynamoDB clients are then initialized with the assume-role credential and region to access resources.
+Here is an example to start Spark shell with this client factory:</p>
+<pre><code class="shell">spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0,software.amazon.awssdk:bundle:2.15.40 \
+ --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
+ --conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
+ --conf spark.sql.catalog.my_catalog.client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory \
+ --conf spark.sql.catalog.my_catalog.client.assume-role.arn=arn:aws:iam::123456789:role/myRoleToAssume \
+ --conf spark.sql.catalog.my_catalog.client.assume-role.region=ap-northeast-1
+</code></pre>
+
+<h2 id="run-iceberg-on-aws">Run Iceberg on AWS<a class="headerlink" href="#run-iceberg-on-aws" title="Permanent link">¶</a></h2>
+<p><a href="https://aws.amazon.com/emr/">Amazon EMR</a> can provision clusters with <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark.html">Spark</a> (EMR 6 for Spark 3, EMR 5 for Spark 2),
+<a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive.html">Hive</a>, <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html">Flink</a>,
+<a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto.html">Trino</a> that can run Iceberg.</p>
+<p><a href="https://aws.amazon.com/about-aws/whats-new/2019/11/you-can-now-run-fully-managed-apache-flink-applications-with-apache-kafka/">Amazon Kinesis Data Analytics</a> provides a platform
+to run fully managed Apache Flink applications. You can include Iceberg in your application Jar and run it in the platform.</p></div>
+
+
+ </div>
+
+ <footer class="col-md-12 text-center">
+
+ <hr>
+ <p>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+
+ <small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
+
+
+
+ </footer>
+ <script src="//ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
+ <script src="../js/bootstrap-3.0.3.min.js"></script>
+ <script src="../js/highlight.pack.js"></script>
+ <script>hljs.initHighlightingOnLoad();</script>
+ <script>var base_url = ".."</script>
+
+ <script src="../js/base.js"></script>
+
+ <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
+ <div class="modal-dialog">
+ <div class="modal-content">
+ <div class="modal-header">
+ <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
+ <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">×</span><span class="sr-only">Close</span></button>
+ </div>
+ <div class="modal-body">
+ <table class="table">
+ <thead>
+ <tr>
+ <th style="width: 20%;">Keys</th>
+ <th>Action</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td class="help shortcut"><kbd>?</kbd></td>
+ <td>Open this help</td>
+ </tr>
+ <tr>
+ <td class="next shortcut"><kbd>n</kbd></td>
+ <td>Next page</td>
+ </tr>
+ <tr>
+ <td class="prev shortcut"><kbd>p</kbd></td>
+ <td>Previous page</td>
+ </tr>
+ <tr>
+ <td class="search shortcut"><kbd>s</kbd></td>
+ <td>Search</td>
+ </tr>
+ </tbody>
+ </table>
+ </div>
+ <div class="modal-footer">
+ </div>
+ </div>
+ </div>
+</div>
+ </body>
+
+</html>
diff --git a/community/index.html b/community/index.html
index 63999a7..34d44db 100644
--- a/community/index.html
+++ b/community/index.html
@@ -71,17 +71,17 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
+<li >
+ <a href="..">About</a>
+</li>
+
+
+
<li class="active">
<a href="./">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -435,7 +450,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/configuration/index.html b/configuration/index.html
index 2ed7b42..6c92096 100644
--- a/configuration/index.html
+++ b/configuration/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -354,6 +369,7 @@
<li class="first-level active"><a href="#configuration">Configuration</a></li>
<li class="second-level"><a href="#catalog-properties">Catalog properties</a></li>
+ <li class="third-level"><a href="#lock-catalog-properties">Lock catalog properties</a></li>
<li class="second-level"><a href="#table-properties">Table properties</a></li>
<li class="third-level"><a href="#read-properties">Read properties</a></li>
@@ -418,12 +434,12 @@
<tr>
<td>uri</td>
<td>null</td>
-<td>(Hive catalog only) the Hive metastore URI</td>
+<td>a URI string, such as Hive metastore URI</td>
</tr>
<tr>
<td>clients</td>
<td>2</td>
-<td>(Hive catalog only) the Hive client pool size</td>
+<td>client pool size</td>
</tr>
</tbody>
</table>
@@ -432,6 +448,49 @@ Any other custom catalog can access the properties by implementing <code>Catalog
The properties can be manually constructed or passed in from a compute engine like Spark or Flink.
Spark uses its session properties as catalog properties, see more details in the <a href="#spark-configuration">Spark configuration</a> section.
Flink passes in catalog properties through <code>CREATE CATALOG</code> statement, see more details in the <a href="../flink/#creating-catalogs-and-using-catalogs">Flink</a> section.</p>
+<h3 id="lock-catalog-properties">Lock catalog properties<a class="headerlink" href="#lock-catalog-properties" title="Permanent link">¶</a></h3>
+<p>Here are the catalog properties related to locking. They are used by some catalog implementations to control the locking behavior during commits.</p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Default</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>lock-impl</td>
+<td>null</td>
+<td>a custom implementation of the lock manager, the actual interface depends on the catalog used</td>
+</tr>
+<tr>
+<td>lock.table</td>
+<td>null</td>
+<td>an auxiliary table for locking, such as in <a href="../aws/#dynamodb-for-commit-locking">AWS DynamoDB lock manager</a></td>
+</tr>
+<tr>
+<td>lock.acquire-interval-ms</td>
+<td>5 seconds</td>
+<td>the interval to wait between each attempt to acquire a lock</td>
+</tr>
+<tr>
+<td>lock.acquire-timeout-ms</td>
+<td>3 minutes</td>
+<td>the maximum time to try acquiring a lock</td>
+</tr>
+<tr>
+<td>lock.heartbeat-interval-ms</td>
+<td>3 seconds</td>
+<td>the interval to wait between each heartbeat after acquiring a lock</td>
+</tr>
+<tr>
+<td>lock.heartbeat-timeout-ms</td>
+<td>15 seconds</td>
+<td>the maximum time without a heartbeat to consider a lock expired</td>
+</tr>
+</tbody>
+</table>
<h2 id="table-properties">Table properties<a class="headerlink" href="#table-properties" title="Permanent link">¶</a></h2>
<p>Iceberg tables support table properties to configure table behavior, like the default split size for readers.</p>
<h3 id="read-properties">Read properties<a class="headerlink" href="#read-properties" title="Permanent link">¶</a></h3>
@@ -537,6 +596,11 @@ Flink passes in catalog properties through <code>CREATE CATALOG</code> statement
<td>Controls the size of files generated to target about this many bytes</td>
</tr>
<tr>
+<td>write.distribution-mode</td>
+<td>none</td>
+<td>Defines distribution of write data: <strong>none</strong>: don’t shuffle rows; <strong>hash</strong>: hash distribute by partition key ; <strong>range</strong>: range distribute by partition key or sort key if table has an SortOrder</td>
+</tr>
+<tr>
<td>write.wap.enabled</td>
<td>false</td>
<td>Enables write-audit-publish writes</td>
@@ -556,6 +620,11 @@ Flink passes in catalog properties through <code>CREATE CATALOG</code> statement
<td>100</td>
<td>The max number of previous version metadata files to keep before deleting after commit</td>
</tr>
+<tr>
+<td>write.spark.fanout.enabled</td>
+<td>false</td>
+<td>Enables Partitioned-Fanout-Writer writes in Spark</td>
+</tr>
</tbody>
</table>
<h3 id="table-behavior-properties">Table behavior properties<a class="headerlink" href="#table-behavior-properties" title="Permanent link">¶</a></h3>
@@ -603,6 +672,16 @@ Flink passes in catalog properties through <code>CREATE CATALOG</code> statement
<td>true</td>
<td>Controls whether to automatically merge manifests on writes</td>
</tr>
+<tr>
+<td>history.expire.max-snapshot-age-ms</td>
+<td>432000000 (5 days)</td>
+<td>Default max age of snapshots to keep while expiring snapshots</td>
+</tr>
+<tr>
+<td>history.expire.min-snapshots-to-keep</td>
+<td>1</td>
+<td>Default min number of snapshots to keep while expiring snapshots</td>
+</tr>
</tbody>
</table>
<h3 id="compatibility-flags">Compatibility flags<a class="headerlink" href="#compatibility-flags" title="Permanent link">¶</a></h3>
@@ -643,8 +722,21 @@ Flink passes in catalog properties through <code>CREATE CATALOG</code> statement
<td>180000 (3 min)</td>
<td>Maximum time in milliseconds to acquire a lock</td>
</tr>
+<tr>
+<td>iceberg.hive.lock-check-min-wait-ms</td>
+<td>50</td>
+<td>Minimum time in milliseconds to check back on the status of lock acquisition</td>
+</tr>
+<tr>
+<td>iceberg.hive.lock-check-max-wait-ms</td>
+<td>5000</td>
+<td>Maximum time in milliseconds to check back on the status of lock acquisition</td>
+</tr>
</tbody>
</table>
+<p>Note: <code>iceberg.hive.lock-check-max-wait-ms</code> should be less than the <a href="https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.txn.timeout">transaction timeout</a>
+of the Hive Metastore (<code>hive.txn.timeout</code> or <code>metastore.txn.timeout</code> in the newer versions). Otherwise, the heartbeats on the lock (which happens during the lock checks) would end up expiring in the
+Hive Metastore before the lock is retried from Iceberg.</p>
<h2 id="spark-configuration">Spark configuration<a class="headerlink" href="#spark-configuration" title="Permanent link">¶</a></h2>
<h3 id="catalogs">Catalogs<a class="headerlink" href="#catalogs" title="Permanent link">¶</a></h3>
<p><a href="../spark/#configuring-catalogs">Spark catalogs</a> are configured using Spark session properties.</p>
@@ -733,6 +825,16 @@ spark.read
<td>As per table property</td>
<td>Overrides this table’s read.split.open-file-cost</td>
</tr>
+<tr>
+<td>vectorization-enabled</td>
+<td>As per table property</td>
+<td>Overrides this table’s read.parquet.vectorization.enabled</td>
+</tr>
+<tr>
+<td>batch-size</td>
+<td>As per table property</td>
+<td>Overrides this table’s read.parquet.vectorization.batch-size</td>
+</tr>
</tbody>
</table>
<h3 id="write-options">Write options<a class="headerlink" href="#write-options" title="Permanent link">¶</a></h3>
@@ -740,6 +842,7 @@ spark.read
<pre><code class="scala">// write with Avro instead of Parquet
df.write
.option("write-format", "avro")
+ .option("snapshot-property.key", "value")
.insertInto("catalog.db.table")
</code></pre>
@@ -772,6 +875,16 @@ df.write
<td>null</td>
<td>Adds an entry with custom-key and corresponding value in the snapshot summary</td>
</tr>
+<tr>
+<td>fanout-enabled</td>
+<td>false</td>
+<td>Overrides this table’s write.spark.fanout.enabled</td>
+</tr>
+<tr>
+<td>check-ordering</td>
+<td>true</td>
+<td>Checks if input schema and table schema are same</td>
+</tr>
</tbody>
</table></div>
@@ -782,7 +895,7 @@ df.write
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/css/extra.css b/css/extra.css
index 4fa7c80..3d79de0 100644
--- a/css/extra.css
+++ b/css/extra.css
@@ -28,6 +28,10 @@
float: left;
}
+.navbar-right {
+ display: none;
+}
+
.navbar-brand {
margin-right: 1em;
}
diff --git a/custom-catalog/index.html b/custom-catalog/index.html
index e1c10e1..8d91a8d 100644
--- a/custom-catalog/index.html
+++ b/custom-catalog/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -212,6 +212,21 @@
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -621,7 +636,7 @@ PARTITIONED BY (category);
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/evolution/index.html b/evolution/index.html
index 42d1610..9fd7a71 100644
--- a/evolution/index.html
+++ b/evolution/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -357,6 +372,8 @@
<li class="third-level"><a href="#correctness">Correctness</a></li>
<li class="second-level"><a href="#partition-evolution">Partition evolution</a></li>
+ <li class="second-level"><a href="#sort-order-evolution">Sort order evolution</a></li>
+
</ul>
</div></div>
<div class="col-md-9" role="main">
@@ -411,7 +428,33 @@
<p><img alt="Partition evolution diagram" src="../img/partition-spec-evolution.png" />
<em>The data for 2008 is partitioned by month. Starting from 2009 the table is updated so that the data is instead partitioned by day. Both partitioning layouts are able to coexist in the same table.</em></p>
<p>Iceberg uses <a href="../partitioning/">hidden partitioning</a>, so you don’t <em>need</em> to write queries for a specific partition layout to be fast. Instead, you can write queries that select the data you need, and Iceberg automatically prunes out files that don’t contain matching data.</p>
-<p>Partition evolution is a metadata operation and does not eagerly rewrite files.</p></div>
+<p>Partition evolution is a metadata operation and does not eagerly rewrite files.</p>
+<p>Iceberg’s Java table API provides <code>updateSpec</code> API to update partition spec.
+For example, the following code could be used to update the partition spec to add a new partition field that places <code>id</code> column values into 8 buckets and remove an existing partition field <code>category</code>:</p>
+<pre><code class="java">Table sampleTable = ...;
+sampleTable.updateSpec()
+ .addField(bucket("id", 8))
+ .removeField("category")
+ .commit();
+</code></pre>
+
+<p>Spark supports updating partition spec through its <code>ALTER TABLE</code> SQL statement, see more details in <a href="../spark/#alter-table-add-partition-field">Spark SQL</a>.</p>
+<h2 id="sort-order-evolution">Sort order evolution<a class="headerlink" href="#sort-order-evolution" title="Permanent link">¶</a></h2>
+<p>Similar to partition spec, Iceberg sort order can also be updated in an existing table.
+When you evolve a sort order, the old data written with an earlier order remains unchanged.
+Engines can always choose to write data in the latest sort order or unsorted when sorting is prohibitively expensive.</p>
+<p>Iceberg’s Java table API provides <code>replaceSortOrder</code> API to update partition spec.
+For example, the following code could be used to create a new sort order
+with <code>id</code> column sorted in ascending order with nulls last,
+and <code>category</code> column sorted in descending order with nulls first:</p>
+<pre><code class="java">Table sampleTable = ...;
+sampleTable.replaceSortOrder()
+ .asc("id", NullOrder.NULLS_LAST)
+ .dec("category", NullOrder.NULL_FIRST)
+ .commit();
+</code></pre>
+
+<p>Spark supports updating sort order through its <code>ALTER TABLE</code> SQL statement, see more details in <a href="../spark/#alter-table-write-ordered-by">Spark SQL</a>.</p></div>
</div>
@@ -420,7 +463,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/flink/index.html b/flink/index.html
index 27dfdf3..674163c 100644
--- a/flink/index.html
+++ b/flink/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -364,23 +379,30 @@
<li class="third-level"><a href="#create-database">CREATE DATABASE</a></li>
<li class="third-level"><a href="#create-table">CREATE TABLE</a></li>
<li class="third-level"><a href="#partitioned-by">PARTITIONED BY</a></li>
+ <li class="third-level"><a href="#create-table-like">CREATE TABLE LIKE</a></li>
<li class="third-level"><a href="#alter-table">ALTER TABLE</a></li>
<li class="third-level"><a href="#alter-table-rename-to">ALTER TABLE .. RENAME TO</a></li>
<li class="third-level"><a href="#drop-table">DROP TABLE</a></li>
<li class="second-level"><a href="#querying-with-sql">Querying with SQL</a></li>
+ <li class="third-level"><a href="#flink-batch-read">Flink batch read</a></li>
+ <li class="third-level"><a href="#flink-streaming-read">Flink streaming read</a></li>
<li class="second-level"><a href="#writing-with-sql">Writing with SQL</a></li>
<li class="third-level"><a href="#insert-into">INSERT INTO</a></li>
<li class="third-level"><a href="#insert-overwrite">INSERT OVERWRITE</a></li>
<li class="second-level"><a href="#reading-with-datastream">Reading with DataStream</a></li>
+ <li class="third-level"><a href="#batch-read">Batch Read</a></li>
+ <li class="third-level"><a href="#streaming-read">Streaming read</a></li>
<li class="second-level"><a href="#writing-with-datastream">Writing with DataStream</a></li>
<li class="third-level"><a href="#appending-data">Appending data.</a></li>
<li class="third-level"><a href="#overwrite-data">Overwrite data</a></li>
<li class="second-level"><a href="#inspecting-tables">Inspecting tables.</a></li>
+ <li class="second-level"><a href="#rewrite-files-action">Rewrite files action.</a></li>
+
<li class="second-level"><a href="#future-improvement">Future improvement.</a></li>
</ul>
@@ -432,6 +454,11 @@ we only integrate iceberg with apache flink 1.11.x .</p>
<td></td>
</tr>
<tr>
+<td><a href="#create-table-like">SQL create table like</a></td>
+<td>✔️</td>
+<td></td>
+</tr>
+<tr>
<td><a href="#alter-table">SQL alter table</a></td>
<td>✔️</td>
<td>Only support altering table properties, Columns/PartitionKey changes are not supported now</td>
@@ -444,7 +471,7 @@ we only integrate iceberg with apache flink 1.11.x .</p>
<tr>
<td><a href="#querying-with-sql">SQL select</a></td>
<td>✔️</td>
-<td>Only support batch mode now.</td>
+<td>Support both streaming and batch mode</td>
</tr>
<tr>
<td><a href="#insert-into">SQL insert into</a></td>
@@ -474,6 +501,11 @@ we only integrate iceberg with apache flink 1.11.x .</p>
<tr>
<td><a href="#inspecting-tables">Metadata tables</a></td>
<td>️</td>
+<td>Support Java API but does not support Flink SQL</td>
+</tr>
+<tr>
+<td><a href="#rewrite-files-action">Rewrite files action</a></td>
+<td>✔️ ️</td>
<td></td>
</tr>
</tbody>
@@ -502,7 +534,7 @@ export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
./bin/sql-client.sh embedded -j <flink-runtime-directory>/iceberg-flink-runtime-xxx.jar shell
</code></pre>
-<p>By default, iceberg has included hadoop jars for hadoop catalog. If we want to use hive catalog, we will need to load the hive jars when opening the flink sql client. Fortunately, apache flink has provided a <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/hive/#using-bundled-hive-jar">bundled hive jar</a> for sql client. So we could open the sql client
+<p>By default, iceberg has included hadoop jars for hadoop catalog. If we want to use hive catalog, we will need to load the hive jars when opening the flink sql client. Fortunately, apache flink has provided a <a href="https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6_2.11/1.11.0/flink-sql-connector-hive-2.3.6_2.11-1.11.0.jar">bundled hive jar</a> for sql client. So we could open the sql client
as the following:</p>
<pre><code class="bash"># HADOOP_HOME is your hadoop root directory after unpack the binary package.
export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
@@ -593,6 +625,17 @@ USE iceberg_db;
</code></pre>
<p>Apache Iceberg support hidden partition but apache flink don’t support partitioning by a function on columns, so we’ve no way to support hidden partition in flink DDL now, we will improve apache flink DDL in future.</p>
+<h3 id="create-table-like"><code>CREATE TABLE LIKE</code><a class="headerlink" href="#create-table-like" title="Permanent link">¶</a></h3>
+<p>To create a table with the same schema, partitioning, and table properties as another table, use <code>CREATE TABLE LIKE</code>.</p>
+<pre><code class="sql">CREATE TABLE hive_catalog.default.sample (
+ id BIGINT COMMENT 'unique id',
+ data STRING
+);
+
+CREATE TABLE hive_catalog.default.sample_like LIKE hive_catalog.default.sample;
+</code></pre>
+
+<p>For more details, refer to the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/create.html#create-table">Flink <code>CREATE TABLE</code> documentation</a>.</p>
<h3 id="alter-table"><code>ALTER TABLE</code><a class="headerlink" href="#alter-table" title="Permanent link">¶</a></h3>
<p>Iceberg only support altering table properties in flink 1.11 now.</p>
<pre><code class="sql">ALTER TABLE hive_catalog.default.sample SET ('write.format.default'='avro')
@@ -608,20 +651,41 @@ USE iceberg_db;
</code></pre>
<h2 id="querying-with-sql">Querying with SQL<a class="headerlink" href="#querying-with-sql" title="Permanent link">¶</a></h2>
-<p>Iceberg does not support streaming read in flink now, it’s still working in-progress. But it support batch read to scan the existing records in iceberg table.</p>
+<p>Iceberg support both streaming and batch read in flink now. we could execute the following sql command to switch the execute type from ‘streaming’ mode to ‘batch’ mode, and vice versa:</p>
<pre><code class="sql">-- Execute the flink job in streaming mode for current session context
+SET execution.type = streaming
+
+-- Execute the flink job in batch mode for current session context
+SET execution.type = batch
+</code></pre>
+
+<h3 id="flink-batch-read">Flink batch read<a class="headerlink" href="#flink-batch-read" title="Permanent link">¶</a></h3>
+<p>If want to check all the rows in iceberg table by submitting a flink <strong>batch</strong> job, you could execute the following sentences:</p>
+<pre><code class="sql">-- Execute the flink job in batch mode for current session context
SET execution.type = batch ;
SELECT * FROM sample ;
</code></pre>
-<p>Notice: we could execute the following sql command to switch the execute type from ‘streaming’ mode to ‘batch’ mode, and vice versa:</p>
-<pre><code class="sql">-- Execute the flink job in streaming mode for current session context
-SET execution.type = streaming
+<h3 id="flink-streaming-read">Flink streaming read<a class="headerlink" href="#flink-streaming-read" title="Permanent link">¶</a></h3>
+<p>Iceberg supports processing incremental data in flink streaming jobs which starts from a historical snapshot-id:</p>
+<pre><code class="sql">-- Submit the flink job in streaming mode for current session.
+SET execution.type = streaming ;
--- Execute the flink job in batch mode for current session context
-SET execution.type = batch
+-- Enable this switch because streaming read SQL will provide few job options in flink SQL hint options.
+SET table.dynamic-table-options.enabled=true;
+
+-- Read all the records from the iceberg current snapshot, and then read incremental data starting from that snapshot.
+SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/ ;
+
+-- Read all incremental data starting from the snapshot-id '3821550127947089987' (records from this snapshot will be excluded).
+SELECT * FROM sample /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s', 'start-snapshot-id'='3821550127947089987')*/ ;
</code></pre>
+<p>Those are the options that could be set in flink SQL hint options for streaming job:</p>
+<ul>
+<li>monitor-interval: time interval for consecutively monitoring newly committed data files (default value: ‘1s’).</li>
+<li>start-snapshot-id: the snapshot id that streaming job starts from.</li>
+</ul>
<h2 id="writing-with-sql">Writing with SQL<a class="headerlink" href="#writing-with-sql" title="Permanent link">¶</a></h2>
<p>Iceberg support both <code>INSERT INTO</code> and <code>INSERT OVERWRITE</code> in flink 1.11 now.</p>
<h3 id="insert-into"><code>INSERT INTO</code><a class="headerlink" href="#insert-into" title="Permanent link">¶</a></h3>
@@ -643,11 +707,47 @@ INSERT INTO hive_catalog.default.sample SELECT id, data from other_kafka_table;
<p>For a partitioned iceberg table, when all the partition columns are set a value in <code>PARTITION</code> clause, it is inserting into a static partition, otherwise if partial partition columns (prefix part of all partition columns) are set a value in <code>PARTITION</code> clause, it is writing the query result into a dynamic partition.
For an unpartitioned iceberg table, its data will be completely overwritten by <code>INSERT OVERWRITE</code>.</p>
<h2 id="reading-with-datastream">Reading with DataStream<a class="headerlink" href="#reading-with-datastream" title="Permanent link">¶</a></h2>
-<p>Iceberg does not support streaming or batch read now, but it’s working in-progress.</p>
+<p>Iceberg support streaming or batch read in Java API now.</p>
+<h3 id="batch-read">Batch Read<a class="headerlink" href="#batch-read" title="Permanent link">¶</a></h3>
+<p>This example will read all records from iceberg table and then print to the stdout console in flink batch job:</p>
+<pre><code class="java">StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
+TableLoader tableLoader = TableLoader.fromHadooptable("hdfs://nn:8020/warehouse/path");
+DataStream<RowData> batch = FlinkSource.forRowData()
+ .env(env)
+ .tableLoader(loader)
+ .streaming(false)
+ .build();
+
+// Print all records to stdout.
+batch.print();
+
+// Submit and execute this batch read job.
+env.execute("Test Iceberg Batch Read");
+</code></pre>
+
+<h3 id="streaming-read">Streaming read<a class="headerlink" href="#streaming-read" title="Permanent link">¶</a></h3>
+<p>This example will read incremental records which start from snapshot-id ‘3821550127947089987’ and print to stdout console in flink streaming job:</p>
+<pre><code class="java">StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
+TableLoader tableLoader = TableLoader.fromHadooptable("hdfs://nn:8020/warehouse/path");
+DataStream<RowData> stream = FlinkSource.forRowData()
+ .env(env)
+ .tableLoader(loader)
+ .streaming(true)
+ .startSnapshotId(3821550127947089987)
+ .build();
+
+// Print all records to stdout.
+stream.print();
+
+// Submit and execute this streaming read job.
+env.execute("Test Iceberg Batch Read");
+</code></pre>
+
+<p>There are other options that we could set by Java API, please see the <a href="./javadoc/master/org/apache/iceberg/flink/source/FlinkSource.html">FlinkSource#Builder</a>.</p>
<h2 id="writing-with-datastream">Writing with DataStream<a class="headerlink" href="#writing-with-datastream" title="Permanent link">¶</a></h2>
<p>Iceberg support writing to iceberg table from different DataStream input.</p>
<h3 id="appending-data">Appending data.<a class="headerlink" href="#appending-data" title="Permanent link">¶</a></h3>
-<p>we have supported writing <code>DataStream<RowData></code> and <code>DataStream<Row</code> to the sink iceberg table natively.</p>
+<p>we have supported writing <code>DataStream<RowData></code> and <code>DataStream<Row></code> to the sink iceberg table natively.</p>
<pre><code class="java">StreamExecutionEnvironment env = ...;
DataStream<RowData> input = ... ;
@@ -682,6 +782,18 @@ env.execute("Test Iceberg DataStream");
<h2 id="inspecting-tables">Inspecting tables.<a class="headerlink" href="#inspecting-tables" title="Permanent link">¶</a></h2>
<p>Iceberg does not support inspecting table in flink sql now, we need to use <a href="../api/">iceberg’s Java API</a> to read iceberg’s meta data to get those table information.</p>
+<h2 id="rewrite-files-action">Rewrite files action.<a class="headerlink" href="#rewrite-files-action" title="Permanent link">¶</a></h2>
+<p>Iceberg provides API to rewrite small files into large files by submitting flink batch job. The behavior of this flink action is the same as the spark’s <a href="./maintenance/#compact-data-files">rewriteDataFiles</a>.</p>
+<pre><code class="java">import org.apache.iceberg.flink.actions.Actions;
+
+TableLoader tableLoader = TableLoader.fromHadooptable("hdfs://nn:8020/warehouse/path");
+Table table = tableLoader.loadTable();
+RewriteDataFilesActionResult result = Actions.forTable(table)
+ .rewriteDataFiles()
+ .execute();
+</code></pre>
+
+<p>For more doc about options of the rewrite files action, please see <a href="./javadoc/master/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html">RewriteDataFilesAction</a></p>
<h2 id="future-improvement">Future improvement.<a class="headerlink" href="#future-improvement" title="Permanent link">¶</a></h2>
<p>There are some features that we do not yet support in the current flink iceberg integration work:</p>
<ul>
@@ -689,7 +801,6 @@ env.execute("Test Iceberg DataStream");
<li>Don’t support creating iceberg table with computed column.</li>
<li>Don’t support creating iceberg table with watermark.</li>
<li>Don’t support adding columns, removing columns, renaming columns, changing columns. <a href="https://issues.apache.org/jira/browse/FLINK-19062">FLINK-19062</a> is tracking this.</li>
-<li>Don’t support flink read iceberg table in streaming mode. <a href="https://github.com/apache/iceberg/issues/1383">#1383</a> is tracking this.</li>
</ul></div>
@@ -699,7 +810,7 @@ env.execute("Test Iceberg DataStream");
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/getting-started/index.html b/getting-started/index.html
index 1cfc8c7..a0b3399 100644
--- a/getting-started/index.html
+++ b/getting-started/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -463,7 +478,7 @@ df.count()
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/hive/index.html b/hive/index.html
index 26e7c85..40d5cd4 100644
--- a/hive/index.html
+++ b/hive/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -337,7 +352,7 @@
</a>
</li>
<li >
- <a rel="next" href="../api/">
+ <a rel="next" href="../aws/">
Next <i class="fas fa-arrow-right"></i>
</a>
</li>
@@ -387,9 +402,9 @@
<pre><code class="sql">add jar /path/to/iceberg-hive-runtime.jar;
</code></pre>
-<p>There are many others ways to achieve this including adding the jar file to Hive’s auxillary classpath (so it is available by default) - please refer to Hive’s documentation for more information.</p>
+<p>There are many others ways to achieve this including adding the jar file to Hive’s auxiliary classpath (so it is available by default) - please refer to Hive’s documentation for more information.</p>
<h4 id="using-hadoop-tables">Using Hadoop Tables<a class="headerlink" href="#using-hadoop-tables" title="Permanent link">¶</a></h4>
-<p>Iceberg tables created using <code>HadoopTables</code> are stored entirely in a directory in a filesytem like HDFS. </p>
+<p>Iceberg tables created using <code>HadoopTables</code> are stored entirely in a directory in a filesystem like HDFS.</p>
<h5 id="create-an-iceberg-table">Create an Iceberg table<a class="headerlink" href="#create-an-iceberg-table" title="Permanent link">¶</a></h5>
<p>The first step is to create an Iceberg table using the Spark/Java/Python API and <code>HadoopTables</code>. For the purposes of this documentation we will assume that the table is called <code>table_a</code> and that the table location is <code>hdfs://some_path/table_a</code>.</p>
<h5 id="create-a-hive-table">Create a Hive table<a class="headerlink" href="#create-a-hive-table" title="Permanent link">¶</a></h5>
@@ -400,7 +415,7 @@ LOCATION 'hdfs://some_bucket/some_path/table_a';
</code></pre>
<h4 id="query-the-iceberg-table-via-hive">Query the Iceberg table via Hive<a class="headerlink" href="#query-the-iceberg-table-via-hive" title="Permanent link">¶</a></h4>
-<p>You should now be able to issue Hive SQL <code>SELECT</code> queries using the above table and see the results returned from the underlying Iceberg table. Both the Map Reduce and Tez query execution engines are supported.</p>
+<p>You should now be able to issue Hive SQL <code>SELECT</code> queries using the above table and see the results returned from the underlying Iceberg table.</p>
<pre><code class="sql">SELECT * from table_a;
</code></pre>
@@ -409,7 +424,7 @@ LOCATION 'hdfs://some_bucket/some_path/table_a';
<h5 id="create-an-iceberg-table_1">Create an Iceberg table<a class="headerlink" href="#create-an-iceberg-table_1" title="Permanent link">¶</a></h5>
<p>The first step is to create an Iceberg table using the Spark/Java/Python API and <code>HiveCatalog</code>. For the purposes of this documentation we will assume that the table is called <code>table_b</code> and that the table location is <code>s3://some_path/table_b</code>. In order for Iceberg to correctly set up the Hive table for querying some configuration values need to be set, the two options for this are described below - you can use either or the other depending on your use case.</p>
<h5 id="hive-configuration">Hive Configuration<a class="headerlink" href="#hive-configuration" title="Permanent link">¶</a></h5>
-<p>The value <code>iceberg.engine.hive.enabled</code> needs to be set to <code>true</code> and added to the Hive configuration file on the classpath of the application creating the table. This can be done by modifying the relevant <code>hive-site.xml</code>. Alternatively this can done programatically like so:</p>
+<p>The value <code>iceberg.engine.hive.enabled</code> needs to be set to <code>true</code> and added to the Hive configuration file on the classpath of the application creating or modifying (altering, inserting etc.) the table. This can be done by modifying the relevant <code>hive-site.xml</code>. Alternatively this can be done programmatically like so:</p>
<pre><code class="java">Configuration hadoopConfiguration = spark.sparkContext().hadoopConfiguration();
hadoopConfiguration.set(ConfigProperties.ENGINE_HIVE_ENABLED, "true"); //iceberg.engine.hive.enabled=true
HiveCatalog catalog = new HiveCatalog(hadoopConfiguration);
@@ -429,13 +444,39 @@ catalog.createTable(tableId, schema, spec);
<pre><code class="sql">SET iceberg.mr.catalog=hive;
</code></pre>
-<p>You should now be able to issue Hive SQL <code>SELECT</code> queries using the above table and see the results returned from the underlying Iceberg table. Both the Map Reduce and Tez query execution engines are supported.</p>
+<p>You should now be able to issue Hive SQL <code>SELECT</code> queries using the above table and see the results returned from the underlying Iceberg table.</p>
<pre><code class="sql">SELECT * from table_b;
</code></pre>
+<h4 id="using-hadoop-catalog">Using Hadoop Catalog<a class="headerlink" href="#using-hadoop-catalog" title="Permanent link">¶</a></h4>
+<p>Iceberg tables created using <code>HadoopCatalog</code> are stored entirely in a directory in a filesystem like HDFS.</p>
+<h5 id="create-an-iceberg-table_2">Create an Iceberg table<a class="headerlink" href="#create-an-iceberg-table_2" title="Permanent link">¶</a></h5>
+<p>The first step is to create an Iceberg table using the Spark/Java/Python API and <code>HadoopCatalog</code>. For the purposes of this documentation we will assume that the fully qualified table identifier is <code>database_a.table_c</code> and that the Hadoop Catalog warehouse location is <code>hdfs://some_bucket/path_to_hadoop_warehouse</code>. Iceberg will therefore create the table at the location <code>hdfs://some_bucket/path_to_hadoop_warehouse/database_a/table_c</code>.</p>
+<h5 id="create-a-hive-table_1">Create a Hive table<a class="headerlink" href="#create-a-hive-table_1" title="Permanent link">¶</a></h5>
+<p>Now overlay a Hive table on top of this Iceberg table by issuing Hive DDL like so:</p>
+<pre><code class="sql">CREATE EXTERNAL TABLE database_a.table_c
+STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
+LOCATION 'hdfs://some_bucket/path_to_hadoop_warehouse/database_a/table_c'
+TBLPROPERTIES (
+ 'iceberg.mr.catalog'='hadoop',
+ 'iceberg.mr.catalog.hadoop.warehouse.location'='hdfs://some_bucket/path_to_hadoop_warehouse')
+;
+</code></pre>
+
+<p>Note that the Hive database and table name <em>must</em> match the values used in the Iceberg <code>TableIdentifier</code> when the table was created. </p>
+<p>It is possible to omit either or both of the table properties but instead you will then need to set these when reading from the table. Generally it is recommended to set them at table creation time so you can query tables created by different catalogs. </p>
+<h4 id="query-the-iceberg-table-via-hive_2">Query the Iceberg table via Hive<a class="headerlink" href="#query-the-iceberg-table-via-hive_2" title="Permanent link">¶</a></h4>
+<p>You should now be able to issue Hive SQL <code>SELECT</code> queries using the above table and see the results returned from the underlying Iceberg table.</p>
+<pre><code class="sql">SELECT * from database_a.table_c;
+</code></pre>
+
<h3 id="features">Features<a class="headerlink" href="#features" title="Permanent link">¶</a></h3>
<h4 id="predicate-pushdown">Predicate pushdown<a class="headerlink" href="#predicate-pushdown" title="Permanent link">¶</a></h4>
-<p>Pushdown of the Hive SQL <code>WHERE</code> clause has been implemented so that these filters are used at the Iceberg TableScan level as well as by the Parquet and ORC Readers.</p></div>
+<p>Pushdown of the Hive SQL <code>WHERE</code> clause has been implemented so that these filters are used at the Iceberg TableScan level as well as by the Parquet and ORC Readers.</p>
+<h4 id="column-projection">Column Projection<a class="headerlink" href="#column-projection" title="Permanent link">¶</a></h4>
+<p>Columns from the Hive SQL <code>SELECT</code> clause are projected down to the Iceberg readers to reduce the number of columns read.</p>
+<h4 id="hive-query-engines">Hive Query Engines<a class="headerlink" href="#hive-query-engines" title="Permanent link">¶</a></h4>
+<p>Both the Map Reduce and Tez query execution engines are supported.</p></div>
</div>
@@ -444,7 +485,7 @@ catalog.createTable(tableId, schema, spec);
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/how-to-release/index.html b/how-to-release/index.html
index 327a244..069f7e0 100644
--- a/how-to-release/index.html
+++ b/how-to-release/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -521,7 +536,7 @@ Thanks to everyone for contributing!
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/index.html b/index.html
index 001646a..1bd6b63 100644
--- a/index.html
+++ b/index.html
@@ -44,7 +44,7 @@
</head>
-<body class="homepage" >
+<body>
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
<div class="container">
@@ -71,17 +71,17 @@
<ul class="nav navbar-nav">
- <li class="active">
- <a href=".">About</a>
- </li>
-
-
-
- <li class="dropdown">
+ <li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
+<li class="active">
+ <a href=".">About</a>
+</li>
+
+
+
<li >
<a href="community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -378,7 +393,7 @@
-->
<h1 id="_1"><img alt="Iceberg" src="img/Iceberg-logo.png" /><a class="headerlink" href="#_1" title="Permanent link">¶</a></h1>
-<p><strong>Apache Iceberg is an open table format for huge analytic datasets.</strong> Iceberg adds tables to Presto and Spark that use a high-performance format that works just like a SQL table.</p>
+<p><strong>Apache Iceberg is an open table format for huge analytic datasets.</strong> Iceberg adds tables to Trino and Spark that use a high-performance format that works just like a SQL table.</p>
<h3 id="user-experience">User experience<a class="headerlink" href="#user-experience" title="Permanent link">¶</a></h3>
<p>Iceberg avoids unpleasant surprises. Schema evolution works and won’t inadvertently un-delete data. Users don’t need to know about partitioning to get fast queries.</p>
<ul>
@@ -411,7 +426,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
@@ -469,8 +484,3 @@
</body>
</html>
-
-<!--
-MkDocs version : 1.0.4
-Build Date UTC : 2020-11-20 01:11:24
--->
diff --git a/java-api-quickstart/index.html b/java-api-quickstart/index.html
index 5d8a743..83c39de 100644
--- a/java-api-quickstart/index.html
+++ b/java-api-quickstart/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -212,6 +212,21 @@
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -440,7 +455,7 @@ Table table = catalog.createTable(name, schema, spec);
import org.apache.iceberg.hadoop.HadoopTables;
import org.apache.iceberg.Table;
-Configuration conf = new Configuration():
+Configuration conf = new Configuration();
HadoopTables tables = new HadoopTables(conf);
Table table = tables.create(schema, spec, table_location);
@@ -514,7 +529,7 @@ PartitionSpec spec = PartitionSpec.builderFor(schema)
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/maintenance/index.html b/maintenance/index.html
index acbd098..20f87f7 100644
--- a/maintenance/index.html
+++ b/maintenance/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -495,7 +510,7 @@ table.rewriteManifests()
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/partitioning/index.html b/partitioning/index.html
index 3fbd69a..c71cdd7 100644
--- a/partitioning/index.html
+++ b/partitioning/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -431,7 +446,9 @@ WHERE event_time BETWEEN '2018-12-01 10:00:00' AND '2018-12-01 12:00:00'
<p>Iceberg produces partition values by taking a column value and optionally transforming it. Iceberg is responsible for converting <code>event_time</code> into <code>event_date</code>, and keeps track of the relationship.</p>
<p>Table partitioning is configured using these relationships. The <code>logs</code> table would be partitioned by <code>date(event_time)</code> and <code>level</code>.</p>
<p>Because Iceberg doesn’t require user-maintained partition columns, it can hide partitioning. Partition values are produced correctly every time and always used to speed up queries, when possible. Producers and consumers wouldn’t even see <code>event_date</code>.</p>
-<p>Most importantly, queries no longer depend on a table’s physical layout. With a separation between physical and logical, Iceberg tables can evolve partition schemes over time as data volume changes. Misconfigured tables can be fixed without an expensive migration.</p></div>
+<p>Most importantly, queries no longer depend on a table’s physical layout. With a separation between physical and logical, Iceberg tables can evolve partition schemes over time as data volume changes. Misconfigured tables can be fixed without an expensive migration.</p>
+<p>For details about all the supported hidden partition transformations, see the <a href="../spec/#partition-transforms">Partition Transforms</a> section.</p>
+<p>For details about updating a table’s partition spec, see the <a href="../evolution/#partition-evolution">partition evolution</a> section.</p></div>
</div>
@@ -440,7 +457,7 @@ WHERE event_time BETWEEN '2018-12-01 10:00:00' AND '2018-12-01 12:00:00'
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/performance/index.html b/performance/index.html
index 5754f75..331ead9 100644
--- a/performance/index.html
+++ b/performance/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -410,7 +425,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/presto/index.html b/presto/index.html
index 1335da8..81d93e5 100644
--- a/presto/index.html
+++ b/presto/index.html
@@ -1,433 +1,11 @@
-<!DOCTYPE html>
-<html lang="en">
+<!doctype html>
+<html lang="en" class="no-js">
<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <meta name="description" content="A table format for large, slow-moving tabular data">
-
-
- <link rel="../img/favicon.ico">
-
-
- <title>Presto - Apache Iceberg</title>
-
-
- <link rel="stylesheet" href="//use.fontawesome.com/releases/v5.5.0/css/all.css" integrity="sha384-B4dIYHKNBt8Bc12p+WXckhzcICo0wtJAoU8YZTY5qE0Id1GSseTk6S+L3BlXeVIU" crossorigin="anonymous">
- <link rel="stylesheet" href="//cdn.jsdelivr.net/npm/hack-font@3.3.0/build/web/hack.min.css">
- <link href='//fonts.googleapis.com/css?family=PT+Sans:400,400italic,700,700italic&subset=latin-ext,latin' rel='stylesheet' type='text/css'>
- <link href='//fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,300,600,700&subset=latin-ext,latin' rel='stylesheet' type='text/css'>
- <link href="../css/bootstrap-custom.min.css" rel="stylesheet">
- <link href="../css/base.min.css" rel="stylesheet">
- <link href="../css/cinder.min.css" rel="stylesheet">
- <link href="../css/highlight.min.css" rel="stylesheet">
- <link href="../css/extra.css" rel="stylesheet">
-
- <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
- <!--[if lt IE 9]>
- <script src="https://cdn.jsdelivr.net/npm/html5shiv@3.7.3/dist/html5shiv.min.js"></script>
- <script src="https://cdn.jsdelivr.net/npm/respond.js@1.4.2/dest/respond.min.js"></script>
- <![endif]-->
-
- <script src="//ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js"></script>
- <script>
- WebFont.load({
- google: {
- families: ['Open Sans', 'PT Sans']
- }
- });
- </script>
-
-
-
-
+ <script>var anchor=window.location.hash.substr(1);location.href="/trino/index.html"+(anchor?"#"+anchor:"")</script>
+ <meta http-equiv="refresh" content="0; url=/trino/index.html">
</head>
-
<body>
-
- <div class="navbar navbar-default navbar-fixed-top" role="navigation">
- <div class="container">
-
- <!-- Collapsed navigation -->
- <div class="navbar-header">
- <!-- Expander button -->
- <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
-
- <!-- Main title -->
-
- <a class="navbar-brand" href="..">Apache Iceberg</a>
- </div>
-
- <!-- Expanded navigation -->
- <div class="navbar-collapse collapse">
- <!-- Main navigation -->
- <ul class="nav navbar-nav">
-
-
- <li >
- <a href="..">About</a>
- </li>
-
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
-<li >
- <a href="../community/">Community</a>
-</li>
-
-
-
-<li >
- <a href="../releases/">Releases</a>
-</li>
-
-
-
-<li >
- <a href="../trademarks/">Trademarks</a>
-</li>
-
-
-
-<li >
- <a href="../how-to-release/">How to Release</a>
-</li>
-
-
- </ul>
- </li>
-
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Tables <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
-<li >
- <a href="../configuration/">Configuration</a>
-</li>
-
-
-
-<li >
- <a href="../schemas/">Schemas</a>
-</li>
-
-
-
-<li >
- <a href="../partitioning/">Partitioning</a>
-</li>
-
-
-
-<li >
- <a href="../evolution/">Table evolution</a>
-</li>
-
-
-
-<li >
- <a href="../maintenance/">Maintenance</a>
-</li>
-
-
-
-<li >
- <a href="../performance/">Performance</a>
-</li>
-
-
-
-<li >
- <a href="../reliability/">Reliability</a>
-</li>
-
-
- </ul>
- </li>
-
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Spark <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
-<li >
- <a href="../getting-started/">Getting Started</a>
-</li>
-
-
-
-<li >
- <a href="../spark/">Spark SQL</a>
-</li>
-
-
-
-<li >
- <a href="../spark-structured-streaming/">Spark Streaming</a>
-</li>
-
-
-
-<li >
- <a href="../spark#time-travel">Time Travel</a>
-</li>
-
-
- </ul>
- </li>
-
-
-
- <li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
- </li>
-
-
-
- <li >
- <a href="../flink/">Flink</a>
- </li>
-
-
-
- <li >
- <a href="../hive/">Hive</a>
- </li>
-
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
-<li >
- <a href="/javadoc/">Javadoc</a>
-</li>
-
-
-
-<li >
- <a href="../api/">Java API intro</a>
-</li>
-
-
-
-<li >
- <a href="../java-api-quickstart/">Java Quickstart</a>
-</li>
-
-
-
-<li >
- <a href="../custom-catalog/">Java Custom Catalog</a>
-</li>
-
-
-
-<li >
- <a href="../python-quickstart/">Python Quickstart</a>
-</li>
-
-
-
-<li >
- <a href="../python-api-intro/">Python API Intro</a>
-</li>
-
-
-
-<li >
- <a href="../python-feature-support/">Python Feature Support</a>
-</li>
-
-
- </ul>
- </li>
-
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Format <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
-<li >
- <a href="../terms/">Definitions</a>
-</li>
-
-
-
-<li >
- <a href="../spec/">Spec</a>
-</li>
-
-
- </ul>
- </li>
-
-
-
- <li >
- <a href="https://github.com/apache/iceberg">GitHub</a>
- </li>
-
-
-
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
-
-<li >
- <a href="https://www.apache.org/licenses/">License</a>
-</li>
-
-
-
-<li >
- <a href="https://www.apache.org/security/">Security</a>
-</li>
-
-
-
-<li >
- <a href="https://www.apache.org/foundation/thanks.html">Sponsors</a>
-</li>
-
-
-
-<li >
- <a href="https://www.apache.org/foundation/sponsorship.html">Donate</a>
-</li>
-
-
-
-<li >
- <a href="https://www.apache.org/events/current-event.html">Events</a>
-</li>
-
-
- </ul>
- </li>
-
-
- </ul>
-
- <ul class="nav navbar-nav navbar-right">
- </ul>
- </div>
- </div>
-</div>
-
- <div class="container">
-
-
- <div class="col-md-3"><div class="bs-sidebar hidden-print affix well" role="complementary">
- <ul class="nav bs-sidenav">
- <li class="first-level active"><a href="#presto">Presto</a></li>
- </ul>
-</div></div>
- <div class="col-md-9" role="main">
-
-<!--
- - Licensed to the Apache Software Foundation (ASF) under one or more
- - contributor license agreements. See the NOTICE file distributed with
- - this work for additional information regarding copyright ownership.
- - The ASF licenses this file to You under the Apache License, Version 2.0
- - (the "License"); you may not use this file except in compliance with
- - the License. You may obtain a copy of the License at
- -
- - http://www.apache.org/licenses/LICENSE-2.0
- -
- - Unless required by applicable law or agreed to in writing, software
- - distributed under the License is distributed on an "AS IS" BASIS,
- - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- - See the License for the specific language governing permissions and
- - limitations under the License.
- -->
-
-<h1 id="presto">Presto<a class="headerlink" href="#presto" title="Permanent link">¶</a></h1>
-<p>The Iceberg connector is available with Presto SQL. It is feature complete and usage details can be found in the <a href="https://prestosql.io/docs/current/connector/iceberg.html">documentation</a></p></div>
-
-
- </div>
-
- <footer class="col-md-12 text-center">
-
- <hr>
- <p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
-
- <small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
-
-
-
- </footer>
- <script src="//ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
- <script src="../js/bootstrap-3.0.3.min.js"></script>
- <script src="../js/highlight.pack.js"></script>
- <script>hljs.initHighlightingOnLoad();</script>
- <script>var base_url = ".."</script>
-
- <script src="../js/base.js"></script>
-
- <div class="modal" id="mkdocs_keyboard_modal" tabindex="-1" role="dialog" aria-labelledby="keyboardModalLabel" aria-hidden="true">
- <div class="modal-dialog">
- <div class="modal-content">
- <div class="modal-header">
- <h4 class="modal-title" id="keyboardModalLabel">Keyboard Shortcuts</h4>
- <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">×</span><span class="sr-only">Close</span></button>
- </div>
- <div class="modal-body">
- <table class="table">
- <thead>
- <tr>
- <th style="width: 20%;">Keys</th>
- <th>Action</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td class="help shortcut"><kbd>?</kbd></td>
- <td>Open this help</td>
- </tr>
- <tr>
- <td class="next shortcut"><kbd>n</kbd></td>
- <td>Next page</td>
- </tr>
- <tr>
- <td class="prev shortcut"><kbd>p</kbd></td>
- <td>Previous page</td>
- </tr>
- <tr>
- <td class="search shortcut"><kbd>s</kbd></td>
- <td>Search</td>
- </tr>
- </tbody>
- </table>
- </div>
- <div class="modal-footer">
- </div>
- </div>
- </div>
-</div>
- </body>
-
+Redirecting...
+</body>
</html>
diff --git a/python-api-intro/index.html b/python-api-intro/index.html
index 13d085c..7943a0e 100644
--- a/python-api-intro/index.html
+++ b/python-api-intro/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -212,6 +212,21 @@
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -488,7 +503,7 @@ list_var = ListType.of_required(1, IntegerType.get());
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/python-feature-support/index.html b/python-feature-support/index.html
index 1fa940a..519a995 100644
--- a/python-feature-support/index.html
+++ b/python-feature-support/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -212,6 +212,21 @@
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -543,7 +558,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/python-quickstart/index.html b/python-quickstart/index.html
index 292a7df..71ff46b 100644
--- a/python-quickstart/index.html
+++ b/python-quickstart/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -212,6 +212,21 @@
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -423,7 +438,7 @@ pprint(int(tbl.current_snapshot().summary.get("total-records")))
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/releases/index.html b/releases/index.html
index bbf9e39..2ccf4e0 100644
--- a/releases/index.html
+++ b/releases/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -484,7 +499,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/reliability/index.html b/reliability/index.html
index 6481b36..304cce9 100644
--- a/reliability/index.html
+++ b/reliability/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -417,7 +432,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/schemas/index.html b/schemas/index.html
index 82f622a..3c7cf2c 100644
--- a/schemas/index.html
+++ b/schemas/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -475,7 +490,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/sitemap.xml b/sitemap.xml
index bcc4634..a96cf41 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,77 +2,77 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -87,12 +87,17 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
+ <changefreq>daily</changefreq>
+ </url>
+ <url>
+ <loc>None</loc>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -102,42 +107,42 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-11-19</lastmod>
+ <lastmod>2021-01-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 6ed28ca..eec72c0 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ
diff --git a/snapshots/index.html b/snapshots/index.html
index f07e0af..cb534c0 100644
--- a/snapshots/index.html
+++ b/snapshots/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -378,7 +393,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/spark-structured-streaming/index.html b/spark-structured-streaming/index.html
index ca83bb3..2f476fe 100644
--- a/spark-structured-streaming/index.html
+++ b/spark-structured-streaming/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -354,6 +369,7 @@
<li class="first-level active"><a href="#spark-structured-streaming">Spark Structured Streaming</a></li>
<li class="second-level"><a href="#streaming-writes">Streaming Writes</a></li>
+ <li class="third-level"><a href="#writing-against-partitioned-table">Writing against partitioned table</a></li>
<li class="second-level"><a href="#maintenance-for-streaming-tables">Maintenance for streaming tables</a></li>
<li class="third-level"><a href="#tune-the-rate-of-commits">Tune the rate of commits</a></li>
@@ -428,6 +444,25 @@ data.writeStream
</ul>
<p>The table should be created in prior to start the streaming query. Refer <a href="/spark/#create-table">SQL create table</a>
on Spark page to see how to create the Iceberg table.</p>
+<h3 id="writing-against-partitioned-table">Writing against partitioned table<a class="headerlink" href="#writing-against-partitioned-table" title="Permanent link">¶</a></h3>
+<p>Iceberg requires the data to be sorted according to the partition spec per task (Spark partition) in prior to write
+against partitioned table. For batch queries you’re encouraged to do explicit sort to fulfill the requirement
+(see <a href="/spark/#writing-against-partitioned-table">here</a>), but the approach would bring additional latency as
+repartition and sort are considered as heavy operations for streaming workload. To avoid additional latency, you can
+enable fanout writer to eliminate the requirement.</p>
+<pre><code class="scala">val tableIdentifier: String = ...
+data.writeStream
+ .format("iceberg")
+ .outputMode("append")
+ .trigger(Trigger.ProcessingTime(1, TimeUnit.MINUTES))
+ .option("path", tableIdentifier)
+ .option("fanout-enabled", "true")
+ .option("checkpointLocation", checkpointPath)
+ .start()
+</code></pre>
+
+<p>Fanout writer opens the files per partition value and doesn’t close these files till write task is finished.
+This functionality is discouraged for batch query, as explicit sort against output rows isn’t expensive for batch workload.</p>
<h2 id="maintenance-for-streaming-tables">Maintenance for streaming tables<a class="headerlink" href="#maintenance-for-streaming-tables" title="Permanent link">¶</a></h2>
<p>Streaming queries can create new table versions quickly, which creates lots of table metadata to track those versions.
Maintaining metadata by tuning the rate of commits, expiring old snapshots, and automatically cleaning up metadata files
@@ -452,7 +487,7 @@ This could lead lots of small manifest files. Manifests can be <a href="../maint
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/spark/index.html b/spark/index.html
index 2fbf2de..0a50d98 100644
--- a/spark/index.html
+++ b/spark/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -369,12 +384,16 @@
<li class="third-level"><a href="#alter-table-rename-column">ALTER TABLE ... RENAME COLUMN</a></li>
<li class="third-level"><a href="#alter-table-alter-column">ALTER TABLE ... ALTER COLUMN</a></li>
<li class="third-level"><a href="#alter-table-drop-column">ALTER TABLE ... DROP COLUMN</a></li>
+ <li class="third-level"><a href="#alter-table-add-partition-field">ALTER TABLE ... ADD PARTITION FIELD</a></li>
+ <li class="third-level"><a href="#alter-table-drop-partition-field">ALTER TABLE ... DROP PARTITION FIELD</a></li>
+ <li class="third-level"><a href="#alter-table-write-ordered-by">ALTER TABLE ... WRITE ORDERED BY</a></li>
<li class="third-level"><a href="#drop-table">DROP TABLE</a></li>
<li class="second-level"><a href="#querying-with-sql">Querying with SQL</a></li>
<li class="second-level"><a href="#querying-with-dataframes">Querying with DataFrames</a></li>
<li class="third-level"><a href="#time-travel">Time travel</a></li>
+ <li class="third-level"><a href="#table-names-and-paths">Table names and paths</a></li>
<li class="third-level"><a href="#spark-24">Spark 2.4</a></li>
<li class="second-level"><a href="#writing-with-sql">Writing with SQL</a></li>
@@ -399,6 +418,18 @@
<li class="third-level"><a href="#spark-type-to-iceberg-type">Spark type to Iceberg type</a></li>
<li class="third-level"><a href="#iceberg-type-to-spark-type">Iceberg type to Spark type</a></li>
+ <li class="second-level"><a href="#procedures">Procedures</a></li>
+
+ <li class="third-level"><a href="#general-usage">General Usage</a></li>
+ <li class="third-level"><a href="#cherrypick-snapshot-procedure">Cherrypick Snapshot Procedure</a></li>
+ <li class="third-level"><a href="#expire-snapshots-procedure">Expire Snapshots Procedure</a></li>
+ <li class="third-level"><a href="#migrate-table-procedure">Migrate Table Procedure</a></li>
+ <li class="third-level"><a href="#remove-orphan-files-procedure">Remove Orphan Files Procedure</a></li>
+ <li class="third-level"><a href="#rewrite-manifests-procedure">Rewrite Manifests Procedure</a></li>
+ <li class="third-level"><a href="#rollback-to-snapshot-procedure">Rollback to Snapshot Procedure</a></li>
+ <li class="third-level"><a href="#rollback-to-timestamp-procedure">Rollback to Timestamp Procedure</a></li>
+ <li class="third-level"><a href="#set-current-snapshot-procedure">Set Current Snapshot Procedure</a></li>
+ <li class="third-level"><a href="#snapshot-table-procedure">Snapshot Table Procedure</a></li>
</ul>
</div></div>
<div class="col-md-9" role="main">
@@ -454,7 +485,7 @@
<td><a href="#alter-table">SQL alter table</a></td>
<td>✔️</td>
<td></td>
-<td></td>
+<td>⚠ requires extensions enabled to update partition field and sort order</td>
</tr>
<tr>
<td><a href="#drop-table">SQL drop table</a></td>
@@ -481,6 +512,12 @@
<td></td>
</tr>
<tr>
+<td><a href="#procedures">SQL procedures</a></td>
+<td>✔️</td>
+<td></td>
+<td>⚠ Requires extensions enabled</td>
+</tr>
+<tr>
<td><a href="#querying-with-dataframes">DataFrame reads</a></td>
<td>✔️</td>
<td>✔️</td>
@@ -512,6 +549,7 @@
</tr>
</tbody>
</table>
+<p>To enable Iceberg SQL extensions, set Spark configuration <code>spark.sql.extensions</code> as <code>org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions</code>. </p>
<h2 id="configuring-catalogs">Configuring catalogs<a class="headerlink" href="#configuring-catalogs" title="Permanent link">¶</a></h2>
<p>Spark 3.0 adds an API to plug in table catalogs that are used to load, create, and manage Iceberg tables. Spark catalogs are configured by setting <a href="../configuration/#catalogs">Spark properties</a> under <code>spark.sql.catalog</code>.</p>
<p>This creates an Iceberg catalog named <code>hive_prod</code> that loads tables from a Hive metastore:</p>
@@ -681,6 +719,41 @@ ALTER TABLE prod.db.sample ALTER COLUMN id COMMENT 'unique id'
ALTER TABLE prod.db.sample DROP COLUMN point.z
</code></pre>
+<h3 id="alter-table-add-partition-field"><code>ALTER TABLE ... ADD PARTITION FIELD</code><a class="headerlink" href="#alter-table-add-partition-field" title="Permanent link">¶</a></h3>
+<pre><code class="sql">ALTER TABLE prod.db.sample ADD PARTITION FIELD catalog -- identity transform
+ALTER TABLE prod.db.sample ADD PARTITION FIELD bucket(16, id)
+ALTER TABLE prod.db.sample ADD PARTITION FIELD truncate(data, 4)
+ALTER TABLE prod.db.sample ADD PARTITION FIELD years(ts)
+-- use optional AS keyword to specify a custom name for the partition field
+ALTER TABLE prod.db.sample ADD PARTITION FIELD bucket(16, id) AS shard
+</code></pre>
+
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>Changing partitioning will change the behavior of dynamic writes, which overwrite any partition that is written to.
+For example, if you partition by days and move to partitioning by hours, overwrites will overwrite hourly partitions but not days anymore.</p>
+</div>
+<h3 id="alter-table-drop-partition-field"><code>ALTER TABLE ... DROP PARTITION FIELD</code><a class="headerlink" href="#alter-table-drop-partition-field" title="Permanent link">¶</a></h3>
+<pre><code class="sql">ALTER TABLE prod.db.sample DROP PARTITION FIELD catalog
+ALTER TABLE prod.db.sample DROP PARTITION FIELD bucket(16, id)
+ALTER TABLE prod.db.sample DROP PARTITION FIELD truncate(data, 4)
+ALTER TABLE prod.db.sample DROP PARTITION FIELD years(ts)
+ALTER TABLE prod.db.sample DROP PARTITION FIELD shard
+</code></pre>
+
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>Changing partitioning will change the behavior of dynamic writes, which overwrite any partition that is written to.
+For example, if you partition by days and move to partitioning by hours, overwrites will overwrite hourly partitions but not days anymore.</p>
+</div>
+<h3 id="alter-table-write-ordered-by"><code>ALTER TABLE ... WRITE ORDERED BY</code><a class="headerlink" href="#alter-table-write-ordered-by" title="Permanent link">¶</a></h3>
+<pre><code class="sql">ALTER TABLE prod.db.sample WRITE ORDERED BY category, id
+-- use optional ASC/DEC keyword to specify sort order of each field (default ASC)
+ALTER TABLE prod.db.sample WRITE ORDERED BY category ASC, id DESC
+-- use optional NULLS FIRST/NULLS LAST keyword to specify null order of each field (default FIRST)
+ALTER TABLE prod.db.sample WRITE ORDERED BY category ASC NULLS LAST, id DESC NULLS FIRST
+</code></pre>
+
<h3 id="drop-table"><code>DROP TABLE</code><a class="headerlink" href="#drop-table" title="Permanent link">¶</a></h3>
<p>To delete a table, run:</p>
<pre><code class="sql">DROP TABLE prod.db.sample
@@ -733,6 +806,19 @@ ignored. Do not use <code>table</code> when attempting to time-travel or use oth
in <a href="https://issues.apache.org/jira/browse/SPARK-32592">Spark 3.1 - SPARK-32592</a>.</p>
</div>
<p>Time travel is not yet supported by Spark’s SQL syntax.</p>
+<h3 id="table-names-and-paths">Table names and paths<a class="headerlink" href="#table-names-and-paths" title="Permanent link">¶</a></h3>
+<p>Paths and table names can be loaded from the Spark3 dataframe interface. How paths/tables are loaded depends on how
+the identifier is specified. When using <code>spark.read().format("iceberg").path(table)</code> or <code>spark.table(table)</code> the <code>table</code>
+variable can take a number of forms as listed below:</p>
+<ul>
+<li><code>file:/path/to/table</code> -> loads a HadoopTable at given path</li>
+<li><code>tablename</code> -> loads <code>currentCatalog.currentNamespace.tablename</code></li>
+<li><code>catalog.tablename</code> -> load <code>tablename</code> from the specified catalog.</li>
+<li><code>namespace.tablename</code> -> load <code>namespace.tablename</code> from current catalog</li>
+<li><code>catalog.namespace.tablename</code> -> load <code>namespace.tablename</code> from the specified catalog.</li>
+<li><code>namespace1.namespace2.tablename</code> -> load <code>namespace1.namespace2.tablename</code> from current catalog</li>
+</ul>
+<p>The above list is in order of priority. For example: a matching catalog will take priority over any namespace resolution.</p>
<h3 id="spark-24">Spark 2.4<a class="headerlink" href="#spark-24" title="Permanent link">¶</a></h3>
<p>Spark 2.4 requires using the DataFrame reader with <code>iceberg</code> as a format, because 2.4 does not support catalogs:</p>
<pre><code class="scala">// named metastore table
@@ -1263,7 +1349,588 @@ so you may want to understand the type conversion in Iceberg in prior to design
<td></td>
</tr>
</tbody>
-</table></div>
+</table>
+<h2 id="procedures">Procedures<a class="headerlink" href="#procedures" title="Permanent link">¶</a></h2>
+<p>In Spark 3 Iceberg provides an SQL API for accomplishing <a href="../maintenance/">maintenance actions</a>. Support for
+stored procedures is tied to the DataSourceV2 catalog and requires that the Iceberg Extensions are enabled for the
+Spark Session.</p>
+<h3 id="general-usage">General Usage<a class="headerlink" href="#general-usage" title="Permanent link">¶</a></h3>
+<p>To call an Iceberg stored procedure, execute a <code>CALL</code> command against the Iceberg catalog. All procedures are added to
+the <code>system</code> keyspace. Procedures can take positional or named arguments.</p>
+<h4 id="generic-call-with-positional-arguments">Generic Call with Positional Arguments<a class="headerlink" href="#generic-call-with-positional-arguments" title="Permanent link">¶</a></h4>
+<pre><code class="sql"> CALL catalog_name.system.procedure_name(arg_1, arg_2, ... arg_n)
+</code></pre>
+
+<h4 id="generic-call-with-named-arguments">Generic Call with Named Arguments<a class="headerlink" href="#generic-call-with-named-arguments" title="Permanent link">¶</a></h4>
+<pre><code class="sql"> CALL catalog_name.system.procedure_name(arg_name_2 => arg_2, arg_name_1 => arg_1)
+</code></pre>
+
+<h3 id="cherrypick-snapshot-procedure">Cherrypick Snapshot Procedure<a class="headerlink" href="#cherrypick-snapshot-procedure" title="Permanent link">¶</a></h3>
+<p>A procedure that applies changes in a given snapshot and creates a new snapshot which will
+be set as the current snapshot in a table.</p>
+<p><strong>Note</strong> this procedure invalidates all cached Spark plans that reference the affected table.</p>
+<h4 id="usage">Usage<a class="headerlink" href="#usage" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to perform the cherrypick on</td>
+</tr>
+<tr>
+<td>snapshot_id</td>
+<td>✔️</td>
+<td>Long</td>
+<td>The snapshot ID to cherrypick</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output">Output<a class="headerlink" href="#output" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>source_snapshot_id</td>
+<td>Long</td>
+<td>The snapshot before applying the cherrypick</td>
+</tr>
+<tr>
+<td>current_snapshot_id</td>
+<td>Long</td>
+<td>The current snapshot ID now that the cherrypick has been applied</td>
+</tr>
+</tbody>
+</table>
+<h4 id="examples">Examples<a class="headerlink" href="#examples" title="Permanent link">¶</a></h4>
+<p>Cherrypick Snapshot 1</p>
+<pre><code class="sql"> CALL catalog_name hive_prod.system.cherrypick_snapshot('my_table', 1)
+</code></pre>
+
+<p>Cherrypick Snapshot 1 with named args</p>
+<pre><code class="sql"> CALL catalog_name hive_prod.system.cherrypick_snapshot(snapshot_id => 1, table => 'my_table' )
+</code></pre>
+
+<h3 id="expire-snapshots-procedure">Expire Snapshots Procedure<a class="headerlink" href="#expire-snapshots-procedure" title="Permanent link">¶</a></h3>
+<p>Each write/update/delete/upsert/compaction in Iceberg produces a new snapshot while keeping the old data and metadata
+around for snapshot isolation and time travel. The <code>expire_snapshots</code> procedure can be used to remove older snapshots
+and their files which are no longer needed.</p>
+<p>This procedure will remove old snapshots and data files which are uniquely required by those old snapshots. This means
+the <code>expire_snapshots</code> procedure will never remove files which are still required by a non-expired snapshot.</p>
+<h4 id="usage_1">Usage<a class="headerlink" href="#usage_1" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to expire snapshots from</td>
+</tr>
+<tr>
+<td>older_than</td>
+<td>️</td>
+<td>Timestamp</td>
+<td>Remove snapshots which were created before this timestamp (Defaults to 5 days ago)</td>
+</tr>
+<tr>
+<td>retain_last</td>
+<td></td>
+<td>Int</td>
+<td>Length of history to preserve regardless of older_than target (defaults to 1)</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_1">Output<a class="headerlink" href="#output_1" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>deleted_data_files_count</td>
+<td>Long</td>
+<td>Number of data files deleted by this operation</td>
+</tr>
+<tr>
+<td>deleted_manifest_files_count</td>
+<td>Long</td>
+<td>Number of manifest files deleted by this operation</td>
+</tr>
+<tr>
+<td>deleted_manifest_lists_count</td>
+<td>Long</td>
+<td>Number of manifest List files deleted by this operation</td>
+</tr>
+</tbody>
+</table>
+<h4 id="examples_1">Examples<a class="headerlink" href="#examples_1" title="Permanent link">¶</a></h4>
+<p>Remove snapshots older than 10 days ago, but retain the last 100 Snapshots.</p>
+<pre><code class="sql"> CALL hive_prod.system.expire_snapshots('db.sample', date_sub(current_date(), 10), 100)
+</code></pre>
+
+<p>Erase all snapshots older than the current timestamp but retain the last 5 snapshots made.</p>
+<pre><code class="sql"> CALL hive_prod.system.expire_snapshots(table => 'db.sample', retain_last => 5)
+</code></pre>
+
+<h3 id="migrate-table-procedure">Migrate Table Procedure<a class="headerlink" href="#migrate-table-procedure" title="Permanent link">¶</a></h3>
+<p>Converts a table known to Spark in this catalog into an Iceberg table. This is mainly for taking
+Hive tables and fully converting them into Iceberg tables but can be used with most Spark Session V1
+tables as well. The migrated table will preserve all properties set on the original table. Once migrated, all
+Iceberg operations are valid.</p>
+<p>If you would like to leave the original table intact and untouched and create new table which shares source
+data files and schema, use the <a href="#snapshot-table-procedure">snapshot procedure</a>.</p>
+<h4 id="usage_2">Usage<a class="headerlink" href="#usage_2" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Table to be migrated to Iceberg</td>
+</tr>
+<tr>
+<td>properties</td>
+<td>️</td>
+<td>Map<String, String></td>
+<td>Table properties to add to the table</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_2">Output<a class="headerlink" href="#output_2" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>migrated_files_count</td>
+<td>Long</td>
+<td>Number of files migrated to Iceberg</td>
+</tr>
+</tbody>
+</table>
+<h4 id="examples_2">Examples<a class="headerlink" href="#examples_2" title="Permanent link">¶</a></h4>
+<p>Migrate the table <code>db.sample</code> to an Iceberg table and add a property ‘foo’ set to ‘bar’:</p>
+<pre><code class="sql"> CALL catalog_name hive_prod.system.migrate('db.sample', map('foo', 'bar'))
+</code></pre>
+
+<p>Migrate the table <code>db.sample</code> to an Iceberg table without adding any additional properties.</p>
+<pre><code class="sql"> CALL catalog_name hive_prod.system.migrate('db.sample')
+</code></pre>
+
+<h3 id="remove-orphan-files-procedure">Remove Orphan Files Procedure<a class="headerlink" href="#remove-orphan-files-procedure" title="Permanent link">¶</a></h3>
+<p>Used for removing files which are no longer
+required by an Iceberg table as they are no longer referenced in any metadata files and can thus be considered “orphaned”.</p>
+<h4 id="usage_3">Usage<a class="headerlink" href="#usage_3" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to remove files from</td>
+</tr>
+<tr>
+<td>older_than</td>
+<td>️</td>
+<td>Timestamp</td>
+<td>Remove orphan files created before this timestamp (Defaults to 3 days ago)</td>
+</tr>
+<tr>
+<td>location</td>
+<td></td>
+<td>String</td>
+<td>Directory to look for files in (defaults to the table’s location)</td>
+</tr>
+<tr>
+<td>dry_run</td>
+<td></td>
+<td>Boolean</td>
+<td>When true, don’t actually remove files (defaults to false)</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_3">Output<a class="headerlink" href="#output_3" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>orphan_file_location</td>
+<td>String</td>
+<td>The path to each file determined to be an orphan by this command</td>
+</tr>
+</tbody>
+</table>
+<h4 id="examples_3">Examples<a class="headerlink" href="#examples_3" title="Permanent link">¶</a></h4>
+<p>Remove any files in the <code>tablelocation/data</code> folder which are not known to the table <code>db.sample</code> and have a timestamp
+older than the current timestamp.</p>
+<pre><code class="sql"> CALL hive_prod.system.remove_orphan_files(table => 'db.sample', older_than => now(), location => 'tablelocation/data')
+</code></pre>
+
+<p>List all the files that are candidates for removal by performing a dry run of the <code>remove_orphan_files</code> command on this table without actually removing them.</p>
+<pre><code class="sql"> CALL hive_prod.system.remove_orphan_files(table => 'db.sample', dry_run => true)
+</code></pre>
+
+<h3 id="rewrite-manifests-procedure">Rewrite Manifests Procedure<a class="headerlink" href="#rewrite-manifests-procedure" title="Permanent link">¶</a></h3>
+<p>A procedure that rewrites manifests in a table and co-locates metadata for partitions. This procedure
+is an optimization which can be used if the number of Manifest files is causing a performance degradation on read.
+Smaller manifest files will be combined into larger ones reducing cost of opening new file handles when performing
+reads.</p>
+<p>See the <a href="./javadoc/master/org/apache/iceberg/actions/RewriteManifestsAction.html"><code>RewriteManifestsAction</code> Javadoc</a>
+to see more configuration options.</p>
+<p><strong>Note</strong> this procedure invalidates all cached Spark plans that reference the affected table.</p>
+<h4 id="usage_4">Usage<a class="headerlink" href="#usage_4" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to rewrite manifests for</td>
+</tr>
+<tr>
+<td>use_caching</td>
+<td>️</td>
+<td>Boolean</td>
+<td>Use Spark caching during operation (defaults to true)</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_4">Output<a class="headerlink" href="#output_4" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>rewritten_manifests_count</td>
+<td>Int</td>
+<td>Number of manifests which were re-written by this command</td>
+</tr>
+<tr>
+<td>added_mainfests_count</td>
+<td>Int</td>
+<td>Number of new manifest files which were written by this command</td>
+</tr>
+</tbody>
+</table>
+<h4 id="examples_4">Examples<a class="headerlink" href="#examples_4" title="Permanent link">¶</a></h4>
+<p>Rewrite the manifests in table <code>db.sample</code> and align manifest files with table partitioning.</p>
+<pre><code class="sql"> CALL hive_prod.system.rewrite_manifests('db.sample')
+</code></pre>
+
+<p>Rewrite the manifests in table <code>db.sample</code> and disable the use of Spark caching. This could be done to
+avoid memory issues on executors.</p>
+<pre><code class="sql"> CALL hive_prod.system.rewrite_manifests('db.sample', false)
+</code></pre>
+
+<h3 id="rollback-to-snapshot-procedure">Rollback to Snapshot Procedure<a class="headerlink" href="#rollback-to-snapshot-procedure" title="Permanent link">¶</a></h3>
+<p>A procedure that rolls a table back to a specific snapshot id. For rolling the table back based on time see
+<a href="#rollback-to-timestamp-procedure">rollback_to_timestamp</a>.</p>
+<p><strong>Note</strong> this procedure invalidates all cached Spark plans that reference the affected table.</p>
+<h4 id="usage_5">Usage<a class="headerlink" href="#usage_5" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to rollback</td>
+</tr>
+<tr>
+<td>snapshot_id</td>
+<td>✔️</td>
+<td>Long</td>
+<td>The snapshot ID to rollback to</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_5">Output<a class="headerlink" href="#output_5" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>previous_snapshot_id</td>
+<td>Long</td>
+<td>The current snapshot ID before the rollback</td>
+</tr>
+<tr>
+<td>current_snapshot_id</td>
+<td>Long</td>
+<td>The new current snapshot ID</td>
+</tr>
+</tbody>
+</table>
+<h4 id="example">Example<a class="headerlink" href="#example" title="Permanent link">¶</a></h4>
+<p>Rollback <code>db.sample</code> to snapshot <code>1</code></p>
+<pre><code class="sql"> CALL hive_prod.system.rollback_to_snapshot('db.sample', 1)
+</code></pre>
+
+<h3 id="rollback-to-timestamp-procedure">Rollback to Timestamp Procedure<a class="headerlink" href="#rollback-to-timestamp-procedure" title="Permanent link">¶</a></h3>
+<p>A procedure that rolls a table back to a certain point in time.</p>
+<p><strong>Note</strong> this procedure invalidates all cached Spark plans that reference the affected table.</p>
+<h4 id="usage_6">Usage<a class="headerlink" href="#usage_6" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to perform rollback on</td>
+</tr>
+<tr>
+<td>timestamp</td>
+<td>✔️</td>
+<td>Timestamp</td>
+<td>The time to rollback to</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_6">Output<a class="headerlink" href="#output_6" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>previous_snapshot_id</td>
+<td>Long</td>
+<td>The current snapshot before rollback</td>
+</tr>
+<tr>
+<td>current_snapshot_id</td>
+<td>Long</td>
+<td>The new current snapshot ID</td>
+</tr>
+</tbody>
+</table>
+<h4 id="example_1">Example<a class="headerlink" href="#example_1" title="Permanent link">¶</a></h4>
+<p>Rollback <code>db.sample</code> to a day ago</p>
+<pre><code class="sql"> CALL hive_prod.system.rollback_to_timestamp('db.sample', date_sub(current_date(), 1))
+</code></pre>
+
+<h3 id="set-current-snapshot-procedure">Set Current Snapshot Procedure<a class="headerlink" href="#set-current-snapshot-procedure" title="Permanent link">¶</a></h3>
+<p>Sets the current snapshot for a table to a different snapshot.</p>
+<p><strong>Note</strong> this procedure invalidates all cached Spark plans that reference the affected table.</p>
+<h4 id="usage_7">Usage<a class="headerlink" href="#usage_7" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of table to change</td>
+</tr>
+<tr>
+<td>snapshot_id</td>
+<td>✔️</td>
+<td>Long</td>
+<td>The snapshot ID to use as current</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_7">Output<a class="headerlink" href="#output_7" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>previous_snapshot_id</td>
+<td>Long</td>
+<td>The snapshot ID before this change</td>
+</tr>
+<tr>
+<td>current_snapshot_id</td>
+<td>Long</td>
+<td>The new current snapshot ID</td>
+</tr>
+</tbody>
+</table>
+<h4 id="example_2">Example<a class="headerlink" href="#example_2" title="Permanent link">¶</a></h4>
+<p>Set the current snapshot for <code>db.sample</code> to 1</p>
+<pre><code class="sql"> CALL hive_prod.system.set_current_snapshot('db.sample', 1)
+</code></pre>
+
+<h3 id="snapshot-table-procedure">Snapshot Table Procedure<a class="headerlink" href="#snapshot-table-procedure" title="Permanent link">¶</a></h3>
+<p>Creates an Iceberg version of a given table without adjusting the underlying table. The newly created table can
+be adjusted or written to without adjusting the underlying table. This procedure could be used to Snapshot a Hive
+production table and produce an Iceberg table referencing the same files. This new table can be used for testing before
+a full migration to Iceberg, or just to see if certain queries would benefit from Iceberg’s optimizations. When
+INSERTS/DELETES and other statements are run on the snapshot, they will create new files in the Snapshot’s location
+rather than the original table location.</p>
+<p><strong>Note</strong> Because tables created by <code>snapshot</code> are not the sole owners of their data files, they are prohibited from
+actions like <code>expire_snapshots</code> which would physically delete data files. Iceberg deletes, which only effect metadata,
+are still allowed. In addition, any operations which affect the original data files will disrupt the Snapshot’s
+integrity. DELETE statements executed against the original Hive table will remove original data files and the
+<code>snapshot</code> table will no longer be able to access them.</p>
+<p>See also <a href="#migrate-table-procedure">migrate table</a> which can be used without disrupting users of the original table.</p>
+<h4 id="usage_8">Usage<a class="headerlink" href="#usage_8" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Argument Name</th>
+<th>Required?</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>source_table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Source table for making the Iceberg table</td>
+</tr>
+<tr>
+<td>table</td>
+<td>✔️</td>
+<td>String</td>
+<td>Name of the new Iceberg table to create</td>
+</tr>
+<tr>
+<td>location</td>
+<td></td>
+<td>String</td>
+<td>Table location for the new table (defaults to the catalog’s default location)</td>
+</tr>
+<tr>
+<td>properties</td>
+<td>️</td>
+<td>Map<String, String></td>
+<td>Properties to add to the newly created table</td>
+</tr>
+</tbody>
+</table>
+<h4 id="output_8">Output<a class="headerlink" href="#output_8" title="Permanent link">¶</a></h4>
+<table>
+<thead>
+<tr>
+<th>Output Name</th>
+<th>Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>imported_files_count</td>
+<td>Long</td>
+<td>Number of files referenced by the new table</td>
+</tr>
+</tbody>
+</table>
+<h4 id="examples_5">Examples<a class="headerlink" href="#examples_5" title="Permanent link">¶</a></h4>
+<p>Make an isolated Iceberg table which references table <code>db.sample</code> named <code>db.snap</code> at the
+catalog’s default location for <code>db.snap</code>.</p>
+<pre><code class="sql"> CALL hive_prod.system.snapshot('db.sample', 'db.snap')
+</code></pre>
+
+<p>Migrate an isolated Iceberg table which references table <code>db.sample</code> named <code>db.snap</code> at
+a manually specified location <code>/tmp/temptable/</code>.</p>
+<pre><code class="sql"> CALL hive_prod.system.snapshot('db.sample', 'db.snap', '/tmp/temptable/')
+</code></pre></div>
</div>
@@ -1272,7 +1939,7 @@ so you may want to understand the type conversion in Iceberg in prior to design
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/spec/index.html b/spec/index.html
index 659127a..e1453ea 100644
--- a/spec/index.html
+++ b/spec/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -369,6 +384,7 @@
<li class="third-level"><a href="#terms">Terms</a></li>
<li class="third-level"><a href="#schemas-and-data-types">Schemas and Data Types</a></li>
<li class="third-level"><a href="#partitioning">Partitioning</a></li>
+ <li class="third-level"><a href="#sorting">Sorting</a></li>
<li class="third-level"><a href="#manifests">Manifests</a></li>
<li class="third-level"><a href="#snapshots">Snapshots</a></li>
<li class="third-level"><a href="#table-metadata">Table Metadata</a></li>
@@ -384,6 +400,7 @@
<li class="third-level"><a href="#schemas">Schemas</a></li>
<li class="third-level"><a href="#partition-specs">Partition Specs</a></li>
+ <li class="third-level"><a href="#sort-orders">Sort Orders</a></li>
<li class="third-level"><a href="#table-metadata-and-snapshots">Table Metadata and Snapshots</a></li>
<li class="second-level"><a href="#appendix-d-single-value-serialization">Appendix D: Single-value serialization</a></li>
@@ -744,6 +761,18 @@
<li>The remainder, <code>v % W</code>, must be positive. For languages where <code>%</code> can produce negative values, the correct truncate function is: <code>v - (((v % W) + W) % W)</code></li>
<li>The width, <code>W</code>, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.</li>
</ol>
+<h3 id="sorting">Sorting<a class="headerlink" href="#sorting" title="Permanent link">¶</a></h3>
+<p>Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a <strong>sort order</strong>.</p>
+<p>A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:</p>
+<ul>
+<li>A <strong>source column id</strong> from the table’s schema</li>
+<li>A <strong>transform</strong> that is used to produce values to be sorted on from the source column. This is the same transform as described in <a href="#partition-transforms">partition transforms</a>.</li>
+<li>A <strong>sort direction</strong>, that can only be either <code>asc</code> or <code>desc</code></li>
+<li>A <strong>null order</strong> that describes the order of null values when sorted. Can only be either <code>nulls-first</code> or <code>nulls-last</code></li>
+</ul>
+<p>Order id <code>0</code> is reserved for the unsorted order. </p>
+<p>Sorting floating-point numbers should produce the following behavior: <code>-NaN</code> < <code>-Infinity</code> < <code>-value</code> < <code>-0</code> < <code>0</code> < <code>value</code> < <code>Infinity</code> < <code>NaN</code>. This aligns with the implementation of Java floating-point types comparisons. </p>
+<p>A data or delete file is associated with a sort order by the sort order’s id within <a href="#manifests">a manifest</a>. Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes.</p>
<h3 id="manifests">Manifests<a class="headerlink" href="#manifests" title="Permanent link">¶</a></h3>
<p>A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a <a href="#snapshots">snapshot</a>, which tracks all of the files in a table at some point in time. Manifests are tracked by a <a href="#manifest-lists">manifest list</a> for each table snapshot.</p>
<p>A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.</p>
@@ -921,7 +950,7 @@
<td><em>optional</em></td>
<td><strong><code>109 value_counts</code></strong></td>
<td><code>map<119: int, 120: long></code></td>
-<td>Map from column id to number of values in the column (including null values)</td>
+<td>Map from column id to number of values in the column (including null and NaN values)</td>
</tr>
<tr>
<td><em>optional</em></td>
@@ -932,6 +961,13 @@
</tr>
<tr>
<td><em>optional</em></td>
+<td><em>optional</em></td>
+<td><strong><code>137 nan_value_counts</code></strong></td>
+<td><code>map<138: int, 139: long></code></td>
+<td>Map from column id to number of NaN values in the column</td>
+</tr>
+<tr>
+<td><em>optional</em></td>
<td></td>
<td><del><strong><code>111 distinct_counts</code></strong></del></td>
<td><code>map<123: int, 124: long></code></td>
@@ -942,14 +978,14 @@
<td><em>optional</em></td>
<td><strong><code>125 lower_bounds</code></strong></td>
<td><code>map<126: int, 127: binary></code></td>
-<td>Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all values in the column for the file</td>
+<td>Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2]</td>
</tr>
<tr>
<td><em>optional</em></td>
<td><em>optional</em></td>
<td><strong><code>128 upper_bounds</code></strong></td>
<td><code>map<129: int, 130: binary></code></td>
-<td>Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all values in the column for the file</td>
+<td>Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2]</td>
</tr>
<tr>
<td><em>optional</em></td>
@@ -972,11 +1008,20 @@
<td><code>list<136: int></code></td>
<td>Field ids used to determine row equality in equality delete files. Required when <code>content=2</code> and should be null otherwise. Fields with ids listed in this column must be present in the delete file</td>
</tr>
+<tr>
+<td><em>optional</em></td>
+<td><em>optional</em></td>
+<td><strong><code>140 sort_order_id</code></strong></td>
+<td><code>int</code></td>
+<td>ID representing sort order for this file [3].</td>
+</tr>
</tbody>
</table>
<p>Notes:</p>
<ol>
<li>Single-value serialization for lower and upper bounds is detailed in Appendix D.</li>
+<li>For <code>float</code> and <code>double</code>, the value <code>-0.0</code> must precede <code>+0.0</code>, as in the IEEE 754 <code>totalOrder</code> predicate.</li>
+<li>If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files should be written with a non-null order id. <a href="#position-delete-files">Position deletes</a> are required to be sorted by file and position, not a table order, and should set sort order id to null. Readers must ignore sort order id for position delete files.</li>
</ol>
<p>The <code>partition</code> struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct’s field ids must match the ids from the partition spec.</p>
<p>The column metrics maps are used when filtering to select both data and delete files. For delete files, the metrics must store bounds and counts for all deleted rows, or must be omitted. Storing metrics for deleted rows ensures that the values can be used during job planning to find delete files that must be merged during a scan.</p>
@@ -1208,20 +1253,21 @@
<td><em>optional</em></td>
<td><strong><code>510 lower_bound</code></strong></td>
<td><code>bytes</code> [1]</td>
-<td>Lower bound for the non-null values in the partition field, or null if all values are null</td>
+<td>Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2]</td>
</tr>
<tr>
<td><em>optional</em></td>
<td><em>optional</em></td>
<td><strong><code>511 upper_bound</code></strong></td>
<td><code>bytes</code> [1]</td>
-<td>Upper bound for the non-null values in the partition field, or null if all values are null</td>
+<td>Upper bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2]</td>
</tr>
</tbody>
</table>
<p>Notes:</p>
<ol>
<li>Lower and upper bounds are serialized to bytes using the single-object serialization in Appendix D. The type of used to encode the value is the type of the partition field data.</li>
+<li>If -0.0 is a value of the partition field, the <code>lower_bound</code> must not be +0.0, and if +0.0 is a value of the partition field, the <code>upper_bound</code> must not be -0.0.</li>
</ol>
<h4 id="scan-planning">Scan Planning<a class="headerlink" href="#scan-planning" title="Permanent link">¶</a></h4>
<p>Scans are planned by reading the manifest files for the current snapshot. Deleted entries in data and delete manifests are not used in a scan.</p>
@@ -1353,6 +1399,18 @@
<td><strong><code>snapshot-log</code></strong></td>
<td>A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed.</td>
</tr>
+<tr>
+<td><em>optional</em></td>
+<td><em>required</em></td>
+<td><strong><code>sort-orders</code></strong></td>
+<td>A list of sort orders, stored as full sort order objects.</td>
+</tr>
+<tr>
+<td><em>optional</em></td>
+<td><em>required</em></td>
+<td><strong><code>default-sort-order-id</code></strong></td>
+<td>Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files.</td>
+</tr>
</tbody>
</table>
<p>For serialization details, see Appendix C.</p>
@@ -2175,6 +2233,68 @@ Hash results are not dependent on decimal scale, which is part of the type, not
</tbody>
</table>
<p>In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated <code>partition-spec</code> field in table metadata. The object format should be used unless otherwise noted in this spec.</p>
+<h3 id="sort-orders">Sort Orders<a class="headerlink" href="#sort-orders" title="Permanent link">¶</a></h3>
+<p>Sort orders are serialized as a list of JSON object, each of which contains the following fields:</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>JSON representation</th>
+<th>Example</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><strong><code>order-id</code></strong></td>
+<td><code>JSON int</code></td>
+<td><code>1</code></td>
+</tr>
+<tr>
+<td><strong><code>fields</code></strong></td>
+<td><code>JSON list: [</code><br /> <code><sort field JSON>,</code><br /> <code>...</code><br /><code>]</code></td>
+<td><code>[ {</code><br /> <code>"transform": "identity",</code><br /> <code>"source-id": 2,</code><br /> <code>"direction": "asc",</code><br /> <code>"null-order": "nulls-first"</code><br /> <code>}, {</code><br /> <code>"transform": "bucket[4]",</code><br /> <code>"source-id": 3,</code><br /> <code>"direction": "desc",</code><br /> <code>"null-order": "nulls-last"</code><br /><code>} ]</c [...]
+</tr>
+</tbody>
+</table>
+<p>Each sort field in the fields list is stored as an object with the following properties:</p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>JSON representation</th>
+<th>Example</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><strong><code>Sort Field</code></strong></td>
+<td><code>JSON object: {</code><br /> <code>"transform": <transform JSON>,</code><br /> <code>"source-id": <source id int>,</code><br /> <code>"direction": <direction string>,</code><br /> <code>"null-order": <null-order string></code><br /><code>}</code></td>
+<td><code>{</code><br /> <code>"transform": "bucket[4]",</code><br /> <code>"source-id": 3,</code><br /> <code>"direction": "desc",</code><br /> <code>"null-order": "nulls-last"</code><br /><code>}</code></td>
+</tr>
+</tbody>
+</table>
+<p>The following table describes the possible values for the some of the field within sort field: </p>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>JSON representation</th>
+<th>Possible values</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><strong><code>direction</code></strong></td>
+<td><code>JSON string</code></td>
+<td><code>"asc", "desc"</code></td>
+</tr>
+<tr>
+<td><strong><code>null-order</code></strong></td>
+<td><code>JSON string</code></td>
+<td><code>"nulls-first", "nulls-last"</code></td>
+</tr>
+</tbody>
+</table>
<h3 id="table-metadata-and-snapshots">Table Metadata and Snapshots<a class="headerlink" href="#table-metadata-and-snapshots" title="Permanent link">¶</a></h3>
<p>Table metadata is serialized as a JSON object according to the following table. Snapshots are not serialized separately. Instead, they are stored in the table metadata JSON.</p>
<table>
@@ -2251,6 +2371,16 @@ Hash results are not dependent on decimal scale, which is part of the type, not
<td><code>JSON list of objects: [</code><br /> <code>{</code><br /> <code>"snapshot-id": ,</code><br /> <code>"timestamp-ms":</code><br /> <code>},</code><br /> <code>...</code><br /><code>]</code></td>
<td><code>[ {</code><br /> <code>"snapshot-id": 30517296...,</code><br /> <code>"timestamp-ms": 1515100...</code><br /><code>} ]</code></td>
</tr>
+<tr>
+<td><strong><code>sort-orders</code></strong></td>
+<td><code>JSON sort orders (list of sort field object)</code></td>
+<td><code>See above</code></td>
+</tr>
+<tr>
+<td><strong><code>default-sort-order-id</code></strong></td>
+<td><code>JSON int</code></td>
+<td><code>0</code></td>
+</tr>
</tbody>
</table>
<h2 id="appendix-d-single-value-serialization">Appendix D: Single-value serialization<a class="headerlink" href="#appendix-d-single-value-serialization" title="Permanent link">¶</a></h2>
@@ -2355,6 +2485,8 @@ Hash results are not dependent on decimal scale, which is part of the type, not
<li>Snapshot added required field field <code>sequence-number</code>.</li>
<li>Snapshot now requires field <code>manifest-list</code>.</li>
<li>Snapshot field <code>manifests</code> is no longer allowed.</li>
+<li>Table metadata now requires field <code>sort-orders</code>.</li>
+<li>Table metadata now requires field <code>default-sort-order-id</code>.</li>
</ul>
<p>Note that these requirements apply when writing data to a v2 table. Tables that are upgraded from v1 may contain metadata that does not follow these requirements. Implementations should remain backward-compatible with v1 metadata requirements.</p></div>
@@ -2365,7 +2497,7 @@ Hash results are not dependent on decimal scale, which is part of the type, not
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/terms/index.html b/terms/index.html
index b9ceac5..e5417d9 100644
--- a/terms/index.html
+++ b/terms/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -414,7 +429,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/trademarks/index.html b/trademarks/index.html
index 176d7a1..f6c4820 100644
--- a/trademarks/index.html
+++ b/trademarks/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown active">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -383,7 +398,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/snapshots/index.html b/trino/index.html
similarity index 91%
copy from snapshots/index.html
copy to trino/index.html
index f07e0af..eb41b99 100644
--- a/snapshots/index.html
+++ b/trino/index.html
@@ -11,7 +11,7 @@
<link rel="../img/favicon.ico">
- <title>Snapshots - Apache Iceberg</title>
+ <title>Trino - Apache Iceberg</title>
<link rel="stylesheet" href="//use.fontawesome.com/releases/v5.5.0/css/all.css" integrity="sha384-B4dIYHKNBt8Bc12p+WXckhzcICo0wtJAoU8YZTY5qE0Id1GSseTk6S+L3BlXeVIU" crossorigin="anonymous">
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -341,11 +356,7 @@
<div class="col-md-3"><div class="bs-sidebar hidden-print affix well" role="complementary">
<ul class="nav bs-sidenav">
- <li class="first-level active"><a href="#snapshots">Snapshots</a></li>
- <li class="second-level"><a href="#time-travel">Time travel</a></li>
-
- <li class="second-level"><a href="#expiration">Expiration</a></li>
-
+ <li class="first-level active"><a href="#trino-formerly-presto-sql">Trino (formerly Presto SQL)</a></li>
</ul>
</div></div>
<div class="col-md-9" role="main">
@@ -367,9 +378,8 @@
- limitations under the License.
-->
-<h1 id="snapshots">Snapshots<a class="headerlink" href="#snapshots" title="Permanent link">¶</a></h1>
-<h2 id="time-travel">Time travel<a class="headerlink" href="#time-travel" title="Permanent link">¶</a></h2>
-<h2 id="expiration">Expiration<a class="headerlink" href="#expiration" title="Permanent link">¶</a></h2></div>
+<h1 id="trino-formerly-presto-sql">Trino (formerly Presto SQL)<a class="headerlink" href="#trino-formerly-presto-sql" title="Permanent link">¶</a></h1>
+<p>The Iceberg connector is available with Trino. It is feature complete and usage details can be found in the <a href="https://trino.io/docs/current/connector/iceberg.html">documentation</a></p></div>
</div>
@@ -378,7 +388,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>
diff --git a/why-iceberg/index.html b/why-iceberg/index.html
index 0cf9fb0..68aa119 100644
--- a/why-iceberg/index.html
+++ b/why-iceberg/index.html
@@ -71,18 +71,18 @@
<ul class="nav navbar-nav">
- <li >
- <a href="..">About</a>
- </li>
-
-
-
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Project <b class="caret"></b></a>
<ul class="dropdown-menu">
<li >
+ <a href="..">About</a>
+</li>
+
+
+
+<li >
<a href="../community/">Community</a>
</li>
@@ -195,7 +195,7 @@
<li >
- <a href="https://prestosql.io/docs/current/connector/iceberg.html">Presto</a>
+ <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>
</li>
@@ -213,6 +213,21 @@
<li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Integrations <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+<li >
+ <a href="../aws/">AWS</a>
+</li>
+
+
+ </ul>
+ </li>
+
+
+
+ <li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API <b class="caret"></b></a>
<ul class="dropdown-menu">
@@ -416,7 +431,7 @@
<hr>
<p>
- <small>Copyright 2018-2020 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
+ <small>Copyright 2018-2021 <a href='https://www.apache.org/'>The Apache Software Foundation</a><br />Apache Iceberg, Iceberg, Apache, the Apache feather logo, and the Apache Iceberg project logo are either registered<br />trademarks or trademarks of The Apache Software Foundation in the United States and other countries.<br></small>
<small>Documentation built with <a href="http://www.mkdocs.org/">MkDocs</a>.</p></small>