You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by gi...@apache.org on 2023/10/06 06:19:40 UTC
[iceberg-docs] branch asf-site updated: deploy: 02496a65c9a23ec862563d4f34e6b5b7390c5bbd
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/iceberg-docs.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bedf64e0 deploy: 02496a65c9a23ec862563d4f34e6b5b7390c5bbd
bedf64e0 is described below
commit bedf64e0ff51e7b2bc3373936a3dd23fcbcaf6b5
Author: nastra <na...@users.noreply.github.com>
AuthorDate: Fri Oct 6 06:19:33 2023 +0000
deploy: 02496a65c9a23ec862563d4f34e6b5b7390c5bbd
---
contribute/index.html | 10 +++++-----
getting-started/index.html | 20 +-------------------
index.html | 2 +-
landingpagesearch.json | 2 +-
spec/index.html | 2 +-
view-spec/index.html | 46 ++++++++++++++++++++++------------------------
6 files changed, 31 insertions(+), 51 deletions(-)
diff --git a/contribute/index.html b/contribute/index.html
index 977e5e80..f5597f64 100644
--- a/contribute/index.html
+++ b/contribute/index.html
@@ -109,13 +109,13 @@ Please refer to the <a href=https://assertj.github.io/doc/#assertj-core-assertio
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>assertThat<span style=color:#f92672>(</span>catalog<span style=color:#f92672>.</span><span style=color:#a6e22e>listNamespaces</span><span style=color:#f92672>()).</span><span style=color:#a6e22e>containsAll</span><span style=color:#f92672>(</span>expected<span style=color:#f92672>);</span>
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#75715e>// ok
</span></span></span><span style=display:flex><span><span style=color:#75715e></span>assertNotNull<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>);</span>
-</span></span><span style=display:flex><span>assertEquals<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>.</span><span style=color:#a6e22e>size</span><span style=color:#f92672>(),</span> 4<span style=color:#f92672>);</span>
+</span></span><span style=display:flex><span>assertEquals<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>.</span><span style=color:#a6e22e>size</span><span style=color:#f92672>(),</span> <span style=color:#ae81ff>4</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#75715e>// better: will show the content of metadataFileLocations if check fails
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span>assertThat<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>).</span><span style=color:#a6e22e>isNotNull</span><span style=color:#f92672>().</span><span style=color:#a6e22e>hasSize</span><span style=color:#f92672>(</span>4<span style=color:#f92672>);</span>
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span>assertThat<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>).</span><span style=color:#a6e22e>isNotNull</span><span style=color:#f92672>().</span><span style=color:#a6e22e>hasSize</span><span style=color:#f92672>(</span><span style=color:#ae81ff>4</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#75715e>// or
-</span></span></span><span style=display:flex><span><span style=color:#75715e></span>assertThat<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>).</span><span style=color:#a6e22e>isNotNull</span><span style=color:#f92672>().</span><span style=color:#a6e22e>hasSameSizeAs</span><span style=color:#f92672>(</span>expected<span style=color:#f92672>).</span><span style=color:#a6e22e>hasSize</span><span style=color:#f92672>(</span>4<span style=color:#f92672>);</span>
+</span></span></span><span style=display:flex><span><span style=color:#75715e></span>assertThat<span style=color:#f92672>(</span>metadataFileLocations<span style=color:#f92672>).</span><span style=color:#a6e22e>isNotNull</span><span style=color:#f92672>().</span><span style=color:#a6e22e>hasSameSizeAs</span><span style=color:#f92672>(</span>expected<span style=color:#f92672>).</span><span style=color:#a6e22e>hasSize</span><span style=color:#f92672>(</span><span style=color:#ae81ff>4</spa [...]
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span><span style=color:#75715e>// bad
</span></span></span><span style=display:flex><span><span style=color:#75715e></span><span style=color:#66d9ef>try</span> <span style=color:#f92672>{</span>
</span></span><span style=display:flex><span> catalog<span style=color:#f92672>.</span><span style=color:#a6e22e>createNamespace</span><span style=color:#f92672>(</span>deniedNamespace<span style=color:#f92672>);</span>
@@ -130,12 +130,12 @@ Please refer to the <a href=https://assertj.github.io/doc/#assertj-core-assertio
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>isInstanceOf</span><span style=color:#f92672>(</span>AccessDeniedException<span style=color:#f92672>.</span><span style=color:#a6e22e>class</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>hasMessage</span><span style=color:#f92672>(</span><span style=color:#e6db74>"User 'testUser' has no permission to create namespace"</span><span style=color:#f92672>);</span>
</span></span></code></pre></div><p>Checks on exceptions should always make sure to assert that a particular exception message has occurred.</p><h3 id=awaitility>Awaitility</h3><p>Avoid using <code>Thread.sleep()</code> in tests as it leads to long test durations and flaky behavior if a condition takes slightly longer than expected.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data- [...]
-</span></span><span style=display:flex><span>Thread<span style=color:#f92672>.</span><span style=color:#a6e22e>sleep</span><span style=color:#f92672>(</span>3000L<span style=color:#f92672>);</span>
+</span></span><span style=display:flex><span>Thread<span style=color:#f92672>.</span><span style=color:#a6e22e>sleep</span><span style=color:#f92672>(</span><span style=color:#ae81ff>3000L</span><span style=color:#f92672>);</span>
</span></span><span style=display:flex><span>assertThat<span style=color:#f92672>(</span>tables<span style=color:#f92672>()).</span><span style=color:#a6e22e>isEmpty</span><span style=color:#f92672>();</span>
</span></span></code></pre></div><p>A better alternative is using <a href=https://github.com/awaitility/awaitility>Awaitility</a> to make sure <code>tables()</code> are eventually empty. The below example will run the check
with a default polling interval of <strong>100 millis</strong>:</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-java data-lang=java><span style=display:flex><span>deleteTablesAsync<span style=color:#f92672>();</span>
</span></span><span style=display:flex><span>Awaitility<span style=color:#f92672>.</span><span style=color:#a6e22e>await</span><span style=color:#f92672>(</span><span style=color:#e6db74>"Tables were not deleted"</span><span style=color:#f92672>)</span>
-</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>atMost</span><span style=color:#f92672>(</span>5<span style=color:#f92672>,</span> TimeUnit<span style=color:#f92672>.</span><span style=color:#a6e22e>SECONDS</span><span style=color:#f92672>)</span>
+</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>atMost</span><span style=color:#f92672>(</span><span style=color:#ae81ff>5</span><span style=color:#f92672>,</span> TimeUnit<span style=color:#f92672>.</span><span style=color:#a6e22e>SECONDS</span><span style=color:#f92672>)</span>
</span></span><span style=display:flex><span> <span style=color:#f92672>.</span><span style=color:#a6e22e>untilAsserted</span><span style=color:#f92672>(()</span> <span style=color:#f92672>-></span> assertThat<span style=color:#f92672>(</span>tables<span style=color:#f92672>()).</span><span style=color:#a6e22e>isEmpty</span><span style=color:#f92672>());</span>
</span></span></code></pre></div><p>Please refer to the <a href=https://github.com/awaitility/awaitility/wiki/Usage>usage guide</a> of <a href=https://github.com/awaitility/awaitility>Awaitility</a> for more usage examples.</p><h3 id=junit4--junit5>JUnit4 / JUnit5</h3><p>Iceberg currently uses a mix of JUnit4 (<code>org.junit</code> imports) and JUnit5 (<code>org.junit.jupiter.api</code> imports) tests. To allow an easier migration to JUnit5 in the future, new test classes
that are being added to the codebase should be written purely in JUnit5 where possible.</p><h2 id=running-benchmarks>Running Benchmarks</h2><p>Some PRs/changesets might require running benchmarks to determine whether they are affecting the baseline performance. Currently there is
diff --git a/getting-started/index.html b/getting-started/index.html
index b3c6e0d8..52cc0335 100644
--- a/getting-started/index.html
+++ b/getting-started/index.html
@@ -1,19 +1 @@
-<!--
- - Licensed to the Apache Software Foundation (ASF) under one or more
- - contributor license agreements. See the NOTICE file distributed with
- - this work for additional information regarding copyright ownership.
- - The ASF licenses this file to You under the Apache License, Version 2.0
- - (the "License"); you may not use this file except in compliance with
- - the License. You may obtain a copy of the License at
- -
- - http://www.apache.org/licenses/LICENSE-2.0
- -
- - Unless required by applicable law or agreed to in writing, software
- - distributed under the License is distributed on an "AS IS" BASIS,
- - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- - See the License for the specific language governing permissions and
- - limitations under the License.
- -->
-<head>
- <meta http-equiv="Refresh" content="0; url='/docs/latest/getting-started'" />
-</head>
+<!doctype html><html lang=en-us><head><title>https://iceberg.apache.org/spark-quickstart/</title><link rel=canonical href=https://iceberg.apache.org/spark-quickstart/><meta name=robots content="noindex"><meta charset=utf-8><meta http-equiv=refresh content="0; url=https://iceberg.apache.org/spark-quickstart/"></head></html>
\ No newline at end of file
diff --git a/index.html b/index.html
index f872a558..9e5a72c5 100644
--- a/index.html
+++ b/index.html
@@ -1,4 +1,4 @@
-<!doctype html><html><head><meta name=generator content="Hugo 0.118.2"><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content><meta name=author content><title>Apache Iceberg</title><link href=/css/bootstrap.css rel=stylesheet><link href=/css/markdown.css rel=stylesheet><link href=/css/katex.min.css rel=stylesheet><link href=/css/iceberg-theme.css rel=stylesheet><link href=/font [...]
+<!doctype html><html><head><meta name=generator content="Hugo 0.119.0"><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content><meta name=author content><title>Apache Iceberg</title><link href=/css/bootstrap.css rel=stylesheet><link href=/css/markdown.css rel=stylesheet><link href=/css/katex.min.css rel=stylesheet><link href=/css/iceberg-theme.css rel=stylesheet><link href=/font [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span>
<span class=icon-bar></span>
diff --git a/landingpagesearch.json b/landingpagesearch.json
index 1c67faa3..27649031 100644
--- a/landingpagesearch.json
+++ b/landingpagesearch.json
@@ -1 +1 @@
-[{"categories":null,"content":" Hive and Iceberg Quickstart This guide will get you up and running with an Iceberg and Hive environment, including sample code to highlight some powerful features. You can learn more about Iceberg’s Hive runtime by checking out the Hive section.\nDocker Images Creating a Table Writing Data to a Table Reading Data from a Table Next Steps Docker Images The fastest way to get started is to use Apache Hive images which provides a SQL-like interface to create a [...]
\ No newline at end of file
+[{"categories":null,"content":" Hive and Iceberg Quickstart This guide will get you up and running with an Iceberg and Hive environment, including sample code to highlight some powerful features. You can learn more about Iceberg’s Hive runtime by checking out the Hive section.\nDocker Images Creating a Table Writing Data to a Table Reading Data from a Table Next Steps Docker Images The fastest way to get started is to use Apache Hive images which provides a SQL-like interface to create a [...]
\ No newline at end of file
diff --git a/spec/index.html b/spec/index.html
index 8be89c19..8fd1ff12 100644
--- a/spec/index.html
+++ b/spec/index.html
@@ -42,7 +42,7 @@ Hash results are not dependent on decimal scale, which is part of the type, not
</span></span><span style=display:flex><span> { <span style=color:#f92672>"field-id"</span>: <span style=color:#ae81ff>4</span>, <span style=color:#f92672>"names"</span>: [<span style=color:#e6db74>"latitude"</span>, <span style=color:#e6db74>"lat"</span>] },
</span></span><span style=display:flex><span> { <span style=color:#f92672>"field-id"</span>: <span style=color:#ae81ff>5</span>, <span style=color:#f92672>"names"</span>: [<span style=color:#e6db74>"longitude"</span>, <span style=color:#e6db74>"long"</span>] }
</span></span><span style=display:flex><span> ] } ]
-</span></span></code></pre></div><h2 id=appendix-d-single-value-serialization>Appendix D: Single-value serialization</h2><h3 id=binary-single-value-serialization>Binary single-value serialization</h3><p>This serialization scheme is for storing single values as individual binary values in the lower and upper bounds maps of manifest files.</p><table><thead><tr><th>Type</th><th>Binary serialization</th></tr></thead><tbody><tr><td><strong><code>boolean</code></strong></td><td><code>0x00</cod [...]
+</span></span></code></pre></div><h3 id=content-file-data-and-delete-serialization>Content File (Data and Delete) Serialization</h3><p>Content file (data or delete) is serialized as a JSON object according to the following table.</p><table><thead><tr><th>Metadata field</th><th>JSON representation</th><th>Example</th></tr></thead><tbody><tr><td><strong><code>spec-id</code></strong></td><td><code>JSON int</code></td><td><code>1</code></td></tr><tr><td><strong><code>content</code></strong>< [...]
<script src=https://iceberg.apache.org//js/jquery.easing.min.js></script>
<script type=text/javascript src=https://iceberg.apache.org//js/search.js></script>
<script src=https://iceberg.apache.org//js/bootstrap.min.js></script>
diff --git a/view-spec/index.html b/view-spec/index.html
index 8e52b706..12ad8d12 100644
--- a/view-spec/index.html
+++ b/view-spec/index.html
@@ -3,7 +3,7 @@
<span class=icon-bar></span>
<span class=icon-bar></span>
<span class=icon-bar></span></button>
-<a class="page-scroll navbar-brand" href=https://iceberg.apache.org/><img class=top-navbar-logo src=https://iceberg.apache.org//img/iceberg-logo-icon.png> Apache Iceberg</a></div><div><input type=search class=form-control id=search-input placeholder=Search... maxlength=64 data-hotkeys=s/></div><div class=versions-dropdown><span>1.3.1</span> <i class="fa fa-chevron-down"></i><div class=versions-dropdown-content><ul><li class=versions-dropdown-selection><a href=/docs/latest>latest</a></li> [...]
+<a class="page-scroll navbar-brand" href=https://iceberg.apache.org/><img class=top-navbar-logo src=https://iceberg.apache.org//img/iceberg-logo-icon.png> Apache Iceberg</a></div><div><input type=search class=form-control id=search-input placeholder=Search... maxlength=64 data-hotkeys=s/></div><div class=versions-dropdown><span>1.3.1</span> <i class="fa fa-chevron-down"></i><div class=versions-dropdown-content><ul><li class=versions-dropdown-selection><a href=/docs/latest>latest</a></li> [...]
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>OR</span> <span style=color:#66d9ef>REPLACE</span> <span style=color:#66d9ef>VIEW</span> event_agg (
</span></span><span style=display:flex><span> event_count <span style=color:#66d9ef>COMMENT</span> <span style=color:#e6db74>'Count of events'</span>,
</span></span><span style=display:flex><span> event_date) <span style=color:#66d9ef>AS</span>
@@ -11,7 +11,7 @@
</span></span><span style=display:flex><span> <span style=color:#66d9ef>COUNT</span>(<span style=color:#ae81ff>1</span>), <span style=color:#66d9ef>CAST</span>(event_ts <span style=color:#66d9ef>AS</span> DATE)
</span></span><span style=display:flex><span><span style=color:#66d9ef>FROM</span> events
</span></span><span style=display:flex><span><span style=color:#66d9ef>GROUP</span> <span style=color:#66d9ef>BY</span> <span style=color:#ae81ff>2</span>
-</span></span></code></pre></div><p>This create statement would produce the following <code>sql</code> representation metadata:</p><table><thead><tr><th>Field name</th><th>Value</th></tr></thead><tbody><tr><td><code>type</code></td><td><code>"sql"</code></td></tr><tr><td><code>sql</code></td><td><code>"SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2"</code></td></tr><tr><td><code>dialect</code></td><td><code>"spark"</code></td></tr><tr><td><code>default-catalog</code>< [...]
+</span></span></code></pre></div><p>This create statement would produce the following <code>sql</code> representation metadata:</p><table><thead><tr><th>Field name</th><th>Value</th></tr></thead><tbody><tr><td><code>type</code></td><td><code>"sql"</code></td></tr><tr><td><code>sql</code></td><td><code>"SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2"</code></td></tr><tr><td><code>dialect</code></td><td><code>"spark"</code></td></tr></tbody></table><p>If a create statem [...]
</span></span></code></pre></div><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-sql data-lang=sql><span style=display:flex><span><span style=color:#66d9ef>CREATE</span> <span style=color:#66d9ef>OR</span> <span style=color:#66d9ef>REPLACE</span> <span style=color:#66d9ef>VIEW</span> event_agg (
</span></span><span style=display:flex><span> event_count <span style=color:#66d9ef>COMMENT</span> <span style=color:#e6db74>'Count of events'</span>,
</span></span><span style=display:flex><span> event_date)
@@ -23,6 +23,7 @@
</span></span><span style=display:flex><span><span style=color:#66d9ef>GROUP</span> <span style=color:#66d9ef>BY</span> <span style=color:#ae81ff>2</span>
</span></span></code></pre></div><p>The metadata JSON file created looks as follows.</p><p>The path is intentionally similar to the path for Iceberg tables and uses a <code>metadata</code> directory.</p><pre tabindex=0><code>s3://bucket/warehouse/default.db/event_agg/metadata/00001-(uuid).metadata.json
</code></pre><pre tabindex=0><code>{
+ "view-uuid": "fa6506c3-7681-40c8-86dc-e36561f83385",
"format-version" : 1,
"location" : "s3://bucket/warehouse/default.db/event_agg",
"current-version-id" : 1,
@@ -33,6 +34,8 @@
"version-id" : 1,
"timestamp-ms" : 1573518431292,
"schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
"summary" : {
"operation" : "create",
"engine-name" : "Spark",
@@ -41,25 +44,21 @@
"representations" : [ {
"type" : "sql",
"sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
- "dialect" : "spark",
- "default-catalog" : "prod",
- "default-namespace" : [ "default" ],
- "field-aliases" : ["event_count", "event_date"],
- "field-comments" : ["Count of events", null]
+ "dialect" : "spark"
} ]
} ],
- "current-schema-id": 1,
"schemas": [ {
"schema-id": 1,
"type" : "struct",
"fields" : [ {
"id" : 1,
- "name" : "col1",
+ "name" : "event_count",
"required" : false,
- "type" : "int"
+ "type" : "int",
+ "doc" : "Count of events"
}, {
"id" : 2,
- "name" : "col2",
+ "name" : "event_date",
"required" : false,
"type" : "date"
} ]
@@ -80,6 +79,7 @@
</span></span><span style=display:flex><span><span style=color:#66d9ef>GROUP</span> <span style=color:#66d9ef>BY</span> <span style=color:#ae81ff>2</span>
</span></span></code></pre></div><p>Updating the view produces a new metadata file that completely replaces the old:</p><pre tabindex=0><code>s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
</code></pre><pre tabindex=0><code>{
+ "view-uuid": "fa6506c3-7681-40c8-86dc-e36561f83385",
"format-version" : 1,
"location" : "s3://bucket/warehouse/default.db/event_agg",
"current-version-id" : 1,
@@ -90,6 +90,8 @@
"version-id" : 1,
"timestamp-ms" : 1573518431292,
"schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
"summary" : {
"operation" : "create",
"engine-name" : "Spark",
@@ -98,15 +100,14 @@
"representations" : [ {
"type" : "sql",
"sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM events\nGROUP BY 2",
- "dialect" : "spark",
- "default-catalog" : "prod",
- "default-namespace" : [ "default" ],
- "field-aliases" : ["event_count", "event_date"],
- "field-comments" : ["Count of events", null]
+ "dialect" : "spark"
} ]
}, {
"version-id" : 2,
"timestamp-ms" : 1573518981593,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
"summary" : {
"operation" : "create",
"engine-name" : "Spark",
@@ -115,24 +116,21 @@
"representations" : [ {
"type" : "sql",
"sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM prod.default.events\nGROUP BY 2",
- "dialect" : "spark",
- "default-catalog" : "prod",
- "default-namespace" : [ "default" ],
- "field-aliases" : ["event_count", "event_date"]
+ "dialect" : "spark"
} ]
} ],
- "current-schema-id": 1,
"schemas": [ {
"schema-id": 1,
"type" : "struct",
"fields" : [ {
"id" : 1,
- "name" : "col1",
+ "name" : "event_count",
"required" : false,
- "type" : "int"
+ "type" : "int",
+ "doc" : "Count of events"
}, {
"id" : 2,
- "name" : "col2",
+ "name" : "event_date",
"required" : false,
"type" : "date"
} ]