You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by bl...@apache.org on 2020/09/16 20:33:46 UTC
[iceberg] branch asf-site updated: Deployed 9856c5f0c with MkDocs
version: 1.0.4
This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 47548eb Deployed 9856c5f0c with MkDocs version: 1.0.4
47548eb is described below
commit 47548eb103519679f819b6835c0e31d6cb3e54f9
Author: Ryan Blue <bl...@apache.org>
AuthorDate: Wed Sep 16 12:33:32 2020 -0800
Deployed 9856c5f0c with MkDocs version: 1.0.4
---
index.html | 2 +-
maintenance/index.html | 8 ++++++++
sitemap.xml | 48 ++++++++++++++++++++++++------------------------
sitemap.xml.gz | Bin 229 -> 229 bytes
spark/index.html | 22 ++++++++++++----------
5 files changed, 45 insertions(+), 35 deletions(-)
diff --git a/index.html b/index.html
index 99c944a..b68449d 100644
--- a/index.html
+++ b/index.html
@@ -472,5 +472,5 @@
<!--
MkDocs version : 1.0.4
-Build Date UTC : 2020-09-14 00:12:03
+Build Date UTC : 2020-09-16 20:33:32
-->
diff --git a/maintenance/index.html b/maintenance/index.html
index 6df8055..acf1345 100644
--- a/maintenance/index.html
+++ b/maintenance/index.html
@@ -452,6 +452,14 @@ Actions.forTable(table)
<p>It is dangerous to remove orphan files with a retention interval shorter than the time expected for any write to complete because it
might corrupt the table if in-progress files are considered orphaned and are deleted. The default interval is 3 days.</p>
</div>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>Iceberg uses the string representations of paths when determining which files need to be removed. On some file systems,
+the path can change over time, but it still represents the same file. For example, if you change authorities for an HDFS cluster,
+none of the old path urls used during creation will match those that appear in a current listing. <em>This will lead to data loss when
+RemoveOrphanFiles is run</em>. Please be sure the entries in your MetadataTables match those listed by the Hadoop
+FileSystem API to avoid unintentional deletion. </p>
+</div>
<h2 id="optional-maintenance">Optional Maintenance<a class="headerlink" href="#optional-maintenance" title="Permanent link">¶</a></h2>
<p>Some tables require additional maintenance. For example, streaming queries may produce small data files that should be <a href="#compact-data-files">compacted into larger files</a>. And some tables can benefit from <a href="#rewrite-manifests">rewriting manifest files</a> to make locating data for queries much faster.</p>
<h3 id="compact-data-files">Compact data files<a class="headerlink" href="#compact-data-files" title="Permanent link">¶</a></h3>
diff --git a/sitemap.xml b/sitemap.xml
index 3e9000e..ec549af 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,82 +2,82 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -92,12 +92,12 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -107,7 +107,7 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -117,27 +117,27 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2020-09-13</lastmod>
+ <lastmod>2020-09-16</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index c2fed7b..94630f9 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ
diff --git a/spark/index.html b/spark/index.html
index aa2d355..e305c73 100644
--- a/spark/index.html
+++ b/spark/index.html
@@ -684,15 +684,9 @@ ALTER TABLE prod.db.sample DROP COLUMN point.z
<pre><code class="scala">val df = spark.table("prod.db.table")
</code></pre>
-<p>To configure the <code>DataFrameReader</code>, use a reader directly:</p>
-<pre><code class="sql">val df = spark.read
- .option("split-size", "268435456")
- .table("prod.db.table")
-</code></pre>
-
<div class="admonition warning">
<p class="admonition-title">Warning</p>
-<p>When reading with DataFrames in Spark 3, use <code>table</code> to load a table by name from a catalog.
+<p>When reading with DataFrames in Spark 3, use <code>table</code> to load a table by name from a catalog unless <code>option</code> is also required.
Using <code>format("iceberg")</code> loads an isolated table reference that is not refreshed when other queries update the table.</p>
</div>
<h3 id="time-travel">Time travel<a class="headerlink" href="#time-travel" title="Permanent link">¶</a></h3>
@@ -704,18 +698,26 @@ Using <code>format("iceberg")</code> loads an isolated table reference that is n
<pre><code class="scala">// time travel to October 26, 1986 at 01:21:00
spark.read
.option("as-of-timestamp", "499162860000")
- .table("prod.db.table")
+ .format("iceberg")
+ .load("path/to/table")
</code></pre>
<pre><code class="scala">// time travel to snapshot with ID 10963874102873L
spark.read
.option("snapshot-id", 10963874102873L)
- .table("prod.db.table")
+ .format("iceberg")
+ .load("path/to/table")
</code></pre>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>Spark does not currently support using <code>option</code> with <code>table</code> in DataFrameReader commands. All options will be silently
+ignored. Do not use <code>table</code> when attempting to time-travel or use other options. Options will be supported with <code>table</code>
+in <a href="https://issues.apache.org/jira/browse/SPARK-32592">Spark 3.1 - SPARK-32592</a>.</p>
+</div>
<p>Time travel is not yet supported by Spark’s SQL syntax.</p>
<h3 id="spark-24">Spark 2.4<a class="headerlink" href="#spark-24" title="Permanent link">¶</a></h3>
-<p>Spark 2.4 requires using the DataFrame reader with <code>iceberg</code> as a format, becuase 2.4 does not support catalogs:</p>
+<p>Spark 2.4 requires using the DataFrame reader with <code>iceberg</code> as a format, because 2.4 does not support catalogs:</p>
<pre><code class="scala">// named metastore table
spark.read.format("iceberg").load("db.table")
// Hadoop path table