You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by bl...@apache.org on 2019/08/24 18:59:36 UTC
[incubator-iceberg] branch asf-site updated: Deployed d071be7f with
MkDocs version: 1.0.4
This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-iceberg.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c1ef33c Deployed d071be7f with MkDocs version: 1.0.4
c1ef33c is described below
commit c1ef33c32e97afffed4cf7158090da52876267ea
Author: Ryan Blue <bl...@apache.org>
AuthorDate: Sat Aug 24 10:59:28 2019 -0800
Deployed d071be7f with MkDocs version: 1.0.4
---
configuration/index.html | 30 +++++++++++++++++
index.html | 2 +-
sitemap.xml | 30 ++++++++---------
sitemap.xml.gz | Bin 216 -> 216 bytes
spark/index.html | 2 +-
spec/index.html | 84 +++++++++++++++++++++++------------------------
6 files changed, 89 insertions(+), 59 deletions(-)
diff --git a/configuration/index.html b/configuration/index.html
index 61f354d..c8048be 100644
--- a/configuration/index.html
+++ b/configuration/index.html
@@ -286,6 +286,11 @@
<td>10</td>
<td>Number of bins to consider when combining input splits</td>
</tr>
+<tr>
+<td>read.split.open-file-cost</td>
+<td>4194304 (4 MB)</td>
+<td>The estimated cost to open a file, used as a minimum weight when combining splits.</td>
+</tr>
</tbody>
</table>
<h3 id="write-properties">Write properties<a class="headerlink" href="#write-properties" title="Permanent link">¶</a></h3>
@@ -333,6 +338,16 @@
<td>none</td>
<td>Metadata compression codec; none or gzip</td>
</tr>
+<tr>
+<td>write.metadata.metrics.default</td>
+<td>truncate(16)</td>
+<td>Default metrics mode for all columns in the table; none, counts, truncate(length), or full</td>
+</tr>
+<tr>
+<td>write.metadata.metrics.column.col1</td>
+<td>(not set)</td>
+<td>Metrics mode for column ‘col1’ to allow per-column tuning; none, counts, truncate(length), or full</td>
+</tr>
</tbody>
</table>
<h3 id="table-behavior-properties">Table behavior properties<a class="headerlink" href="#table-behavior-properties" title="Permanent link">¶</a></h3>
@@ -406,6 +421,21 @@ spark.read
<td>(latest)</td>
<td>A timestamp in milliseconds; the snapshot used will be the snapshot current at this time.</td>
</tr>
+<tr>
+<td>split-size</td>
+<td>As per table property</td>
+<td>Overrides this table’s read.split.target-size</td>
+</tr>
+<tr>
+<td>lookback</td>
+<td>As per table property</td>
+<td>Overrides this table’s read.split.planning-lookback</td>
+</tr>
+<tr>
+<td>file-open-cost</td>
+<td>As per table property</td>
+<td>Overrides this table’s read.split.open-file-cost</td>
+</tr>
</tbody>
</table>
<h3 id="write-options">Write options<a class="headerlink" href="#write-options" title="Permanent link">¶</a></h3>
diff --git a/index.html b/index.html
index dde4213..a3af56c 100644
--- a/index.html
+++ b/index.html
@@ -368,5 +368,5 @@
<!--
MkDocs version : 1.0.4
-Build Date UTC : 2019-07-15 04:40:39
+Build Date UTC : 2019-08-24 18:59:28
-->
diff --git a/sitemap.xml b/sitemap.xml
index 55f7197..0ae974f 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,42 +2,42 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -47,17 +47,17 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -67,12 +67,12 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
@@ -87,12 +87,12 @@
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>None</loc>
- <lastmod>2019-07-14</lastmod>
+ <lastmod>2019-08-24</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index ea50037..4107ede 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ
diff --git a/spark/index.html b/spark/index.html
index 743ed2c..ac9563d 100644
--- a/spark/index.html
+++ b/spark/index.html
@@ -354,7 +354,7 @@
</div>
<h2 id="spark-24">Spark 2.4<a class="headerlink" href="#spark-24" title="Permanent link">¶</a></h2>
<p>To use Iceberg in Spark 2.4, add the <code>iceberg-spark-runtime</code> Jar to Spark’s <code>jars</code> folder.</p>
-<p>Spark 2.4 is limited to reading and writing existing Iceberg tables. Use the <a href="api">Iceberg API</a> to create Iceberg tables.</p>
+<p>Spark 2.4 is limited to reading and writing existing Iceberg tables. Use the <a href="../api">Iceberg API</a> to create Iceberg tables.</p>
<h3 id="reading-an-iceberg-table">Reading an Iceberg table<a class="headerlink" href="#reading-an-iceberg-table" title="Permanent link">¶</a></h3>
<p>To read an Iceberg table, use the <code>iceberg</code> format in <code>DataFrameReader</code>:</p>
<pre><code class="scala">spark.read.format("iceberg").load("db.table")
diff --git a/spec/index.html b/spec/index.html
index 4e68733..5d906bc 100644
--- a/spec/index.html
+++ b/spec/index.html
@@ -307,21 +307,21 @@
<h4 id="file-system-operations">File System Operations<a class="headerlink" href="#file-system-operations" title="Permanent link">¶</a></h4>
<p>Iceberg only requires that file systems support the following operations:</p>
<ul>
-<li><strong>In-place write</strong>: files are not moved or altered once they are written</li>
-<li><strong>Seekable reads</strong>: data file formats require seek support</li>
-<li><strong>Deletes</strong>: tables delete files that are no longer used</li>
+<li><strong>In-place write</strong> – Files are not moved or altered once they are written.</li>
+<li><strong>Seekable reads</strong> – Data file formats require seek support.</li>
+<li><strong>Deletes</strong> – Tables delete files that are no longer used.</li>
</ul>
<p>These requirements are compatible with object stores, like S3.</p>
<p>Tables do not require random-access writes. Once written, data and metadata files are immutable until they are deleted.</p>
-<p>Tables do not require rename, except fo rtables that use atomic rename to implement the commit operation for new metadata files.</p>
+<p>Tables do not require rename, except for tables that use atomic rename to implement the commit operation for new metadata files.</p>
<h2 id="specification">Specification<a class="headerlink" href="#specification" title="Permanent link">¶</a></h2>
<h4 id="terms">Terms<a class="headerlink" href="#terms" title="Permanent link">¶</a></h4>
<ul>
-<li><strong>Schema</strong> – names and types of fields in a table</li>
-<li><strong>Partition spec</strong> – a definition of how partition values are derived from data fields</li>
-<li><strong>Snapshot</strong> – the state of a table at some point in time, including the set of all data files</li>
-<li><strong>Manifest</strong> – a file that lists data files; a subset of a snapshot</li>
-<li><strong>Manifest list</strong> – a file that lists manifest files; one per snapshot</li>
+<li><strong>Schema</strong> – Names and types of fields in a table.</li>
+<li><strong>Partition spec</strong> – A definition of how partition values are derived from data fields.</li>
+<li><strong>Snapshot</strong> – The state of a table at some point in time, including the set of all data files.</li>
+<li><strong>Manifest</strong> – A file that lists data files; a subset of a snapshot.</li>
+<li><strong>Manifest list</strong> – A file that lists manifest files; one per snapshot.</li>
</ul>
<h3 id="schemas-and-data-types">Schemas and Data Types<a class="headerlink" href="#schemas-and-data-types" title="Permanent link">¶</a></h3>
<p>A table’s <strong>schema</strong> is a list of named columns. All data types are either primitives or nested types, which are maps, lists, or structs. A table schema is also a struct type.</p>
@@ -691,16 +691,16 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<h3 id="snapshots">Snapshots<a class="headerlink" href="#snapshots" title="Permanent link">¶</a></h3>
<p>A snapshot consists of the following fields:</p>
<ul>
-<li><strong><code>snapshot-id</code></strong>: a unique long ID.</li>
-<li><strong><code>parent-snapshot-id</code></strong>: (optional) the snapshot ID of the snapshot’s parent. This field is not present for snapshots that have no parent snapshot, such as snapshots created before this field was added or the first snapshot of a table.</li>
-<li><strong><code>timestamp-ms</code></strong>: a timestamp when the snapshot was created. This is used when garbage collecting snapshots.</li>
-<li><strong><code>manifests</code></strong>: a list of manifest file locations. The data files in a snapshot are the union of all data files listed in these manifests. (Deprecated in favor of <code>manifest-list</code>)</li>
-<li><strong><code>manifest-list</code></strong>: (optional) the location of a manifest list file for this snapshot, which contains a list of manifest files with additional metadata. If present, the manifests field must be omitted.</li>
-<li><strong><code>summary</code></strong>: (optional) a summary that encodes the <code>operation</code> that produced the snapshot and other relevant information specific to that operation. This allows some operations like snapshot expiration to skip processing some snapshots. Possible values of <code>operation</code> are:<ul>
-<li><code>append</code>: data files were added and no files were removed.</li>
-<li><code>replace</code>: data files were rewritten with the same data; i.e., compaction, changing the data file format, or relocating data files.</li>
-<li><code>overwrite</code>: data files were deleted and added in a logical overwrite operation.</li>
-<li><code>delete</code>: data files were removed and their contents logically deleted.</li>
+<li><strong><code>snapshot-id</code></strong> – A unique long ID.</li>
+<li><strong><code>parent-snapshot-id</code></strong> – (Optional) The snapshot ID of the snapshot’s parent. This field is not present for snapshots that have no parent snapshot, such as snapshots created before this field was added or the first snapshot of a table.</li>
+<li><strong><code>timestamp-ms</code></strong> – A timestamp when the snapshot was created. This is used when garbage collecting snapshots.</li>
+<li><strong><code>manifests</code></strong> – A list of manifest file locations. The data files in a snapshot are the union of all data files listed in these manifests. (Deprecated in favor of <code>manifest-list</code>)</li>
+<li><strong><code>manifest-list</code></strong> – (Optional) The location of a manifest list file for this snapshot, which contains a list of manifest files with additional metadata. If present, the manifests field must be omitted.</li>
+<li><strong><code>summary</code></strong> – (Optional) A summary that encodes the <code>operation</code> that produced the snapshot and other relevant information specific to that operation. This allows some operations like snapshot expiration to skip processing some snapshots. Possible values of <code>operation</code> are:<ul>
+<li><code>append</code> – Data files were added and no files were removed.</li>
+<li><code>replace</code> – Data files were rewritten with the same data; i.e., compaction, changing the data file format, or relocating data files.</li>
+<li><code>overwrite</code> – Data files were deleted and added in a logical overwrite operation.</li>
+<li><code>delete</code> – Data files were removed and their contents logically deleted.</li>
</ul>
</li>
</ul>
@@ -713,7 +713,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<p>Valid snapshots are stored as a list in table metadata. For serialization, see Appendix C.</p>
<h4 id="scan-planning">Scan Planning<a class="headerlink" href="#scan-planning" title="Permanent link">¶</a></h4>
<p>Scans are planned by reading the manifest files for the current snapshot listed in the table metadata. Deleted entries in a manifest are not included in the scan.</p>
-<p>For each manifest, scan predicates, that filter data rows, are converted to partition predicates, that filter data files, and used to select the data files in the manifest. This conversion uses the partition spec used to write the manifest file.</p>
+<p>For each manifest, scan predicates, which filter data rows, are converted to partition predicates, which filter data files. These partition predicates are used select the data files in the manifest. This conversion uses the partition spec used to write the manifest file.</p>
<p>Scan predicates are converted to partition predicates using an inclusive projection: if a scan predicate matches a row, then the partition predicate must match that row’s partition. This is an <em>inclusive projection</em> [1] because rows that do not match the scan predicate may be included in the scan by the partition predicate.</p>
<p>For example, an <code>events</code> table with a timestamp column named <code>ts</code> that is partitioned by <code>ts_day=day(ts)</code> is queried by users with ranges over the timestamp column: <code>ts > X</code>. The inclusive projection is <code>ts_day >= day(X)</code>, which is used to select files that may have matching rows. Note that, in most cases, timestamps just before <code>X</code> will be included in the scan because the file contains rows that match the predica [...]
<p>Notes:</p>
@@ -788,7 +788,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>509 contains_null</code></strong></td>
<td><code>boolean</code></td>
-<td>Whether the manifest contains at least one partition with a null value for the field</td>
+<td>Whether the manifest contains at least one partition with a null value for the field.</td>
</tr>
<tr>
<td><strong><code>510 lower_bound</code></strong></td>
@@ -964,22 +964,22 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>date</code></strong></td>
<td><code>{ "type": "int",</code><br /> <code>"logicalType": "date" }</code></td>
-<td>Stores days from the 1970-01-01</td>
+<td>Stores days from the 1970-01-01.</td>
</tr>
<tr>
<td><strong><code>time</code></strong></td>
<td><code>{ "type": "long",</code><br /> <code>"logicalType": "time-micros" }</code></td>
-<td>Stores microseconds from midnight</td>
+<td>Stores microseconds from midnight.</td>
</tr>
<tr>
<td><strong><code>timestamp</code></strong></td>
<td><code>{ "type": "long",</code><br /> <code>"logicalType": "timestamp-micros",</code><br /> <code>"adjust-to-utc": false }</code></td>
-<td>Stores microseconds from 1970-01-01 00:00:00.000000</td>
+<td>Stores microseconds from 1970-01-01 00:00:00.000000.</td>
</tr>
<tr>
<td><strong><code>timestamptz</code></strong></td>
<td><code>{ "type": "long",</code><br /> <code>"logicalType": "timestamp-micros",</code><br /> <code>"adjust-to-utc": true }</code></td>
-<td>Stores microseconds from 1970-01-01 00:00:00.000000 UTC</td>
+<td>Stores microseconds from 1970-01-01 00:00:00.000000 UTC.</td>
</tr>
<tr>
<td><strong><code>string</code></strong></td>
@@ -1013,7 +1013,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
</tr>
<tr>
<td><strong><code>map</code></strong></td>
-<td><code>array</code> of key-value records, or <code>map</code> when keys are strings (optional)</td>
+<td><code>array</code> of key-value records, or <code>map</code> when keys are strings (optional).</td>
<td>Array storage must use logical type name <code>map</code> and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.</td>
</tr>
</tbody>
@@ -1118,31 +1118,31 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<td><strong><code>date</code></strong></td>
<td><code>int32</code></td>
<td><code>DATE</code></td>
-<td>Stores days from the 1970-01-01</td>
+<td>Stores days from the 1970-01-01.</td>
</tr>
<tr>
<td><strong><code>time</code></strong></td>
<td><code>int64</code></td>
<td><code>TIME_MICROS</code> with <code>adjustToUtc=false</code></td>
-<td>Stores microseconds from midnight</td>
+<td>Stores microseconds from midnight.</td>
</tr>
<tr>
<td><strong><code>timestamp</code></strong></td>
<td><code>int64</code></td>
<td><code>TIMESTAMP_MICROS</code> with <code>adjustToUtc=false</code></td>
-<td>Stores microseconds from 1970-01-01 00:00:00.000000</td>
+<td>Stores microseconds from 1970-01-01 00:00:00.000000.</td>
</tr>
<tr>
<td><strong><code>timestamptz</code></strong></td>
<td><code>int64</code></td>
<td><code>TIMESTAMP_MICROS</code> with <code>adjustToUtc=true</code></td>
-<td>Stores microseconds from 1970-01-01 00:00:00.000000 UTC</td>
+<td>Stores microseconds from 1970-01-01 00:00:00.000000 UTC.</td>
</tr>
<tr>
<td><strong><code>string</code></strong></td>
<td><code>binary</code></td>
<td><code>UTF8</code></td>
-<td>Encoding must be UTF-8</td>
+<td>Encoding must be UTF-8.</td>
</tr>
<tr>
<td><strong><code>uuid</code></strong></td>
@@ -1172,13 +1172,13 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<td><strong><code>list</code></strong></td>
<td><code>3-level list</code></td>
<td><code>LIST</code></td>
-<td>See Parquet docs for 3-level representation</td>
+<td>See Parquet docs for 3-level representation.</td>
</tr>
<tr>
<td><strong><code>map</code></strong></td>
<td><code>3-level map</code></td>
<td><code>MAP</code></td>
-<td>See Parquet docs for 3-level representation</td>
+<td>See Parquet docs for 3-level representation.</td>
</tr>
</tbody>
</table>
@@ -1201,7 +1201,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>int</code></strong></td>
<td><code>int</code></td>
-<td>ORC tinyint and smallint would map to int also.</td>
+<td>ORC <code>tinyint</code> and <code>smallint</code> would also map to <strong><code>int</code></strong>.</td>
</tr>
<tr>
<td><strong><code>long</code></strong></td>
@@ -1231,7 +1231,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>time</code></strong></td>
<td><code>int</code></td>
-<td>Stores microseconds from midnight</td>
+<td>Stores microseconds from midnight.</td>
</tr>
<tr>
<td><strong><code>timestamp</code></strong></td>
@@ -1241,12 +1241,12 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>timestamptz</code></strong></td>
<td><code>struct</code></td>
-<td>We should add this to ORC’s type model. (ORC-294)</td>
+<td>We should add this to ORC’s type model (ORC-294).</td>
</tr>
<tr>
<td><strong><code>string</code></strong></td>
<td><code>string</code></td>
-<td>ORC varchar and char would map to Iceberg string too.</td>
+<td>ORC <code>varchar</code> and <code>char</code> would also map to <strong><code>string</code></strong>.</td>
</tr>
<tr>
<td><strong><code>uuid</code></strong></td>
@@ -1256,7 +1256,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>fixed(L)</code></strong></td>
<td><code>binary</code></td>
-<td>The length would not be checked by the ORC reader and should be checked by the adaptor.</td>
+<td>The length would not be checked by the ORC reader and should be checked by the adapter.</td>
</tr>
<tr>
<td><strong><code>binary</code></strong></td>
@@ -1266,7 +1266,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
<tr>
<td><strong><code>struct</code></strong></td>
<td><code>struct</code></td>
-<td>ORC uniontype would map to struct also.</td>
+<td>ORC <code>uniontype</code> would also map to <strong><code>struct</code></strong>.</td>
</tr>
<tr>
<td><strong><code>list</code></strong></td>
@@ -1280,8 +1280,8 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
</tr>
</tbody>
</table>
-<p>One of the interesting challenges with this is how to map Iceberg’s schema evolution (id based) on to ORC’s (name based). In theory we could use Iceberg’s column ids as the column and field names, but that would suck from a user’s point of view. </p>
-<p>The column ids would be stored in ORC’s user metadata as “iceberg.column.id” with a comma separated list of the ids.</p>
+<p>One of the interesting challenges with this is how to map Iceberg’s schema evolution (id based) on to ORC’s (name based). In theory, we could use Iceberg’s column ids as the column and field names, but that would suck from a user’s point of view. </p>
+<p>The column ids would be stored in ORC’s user metadata as <code>iceberg.column.id</code> with a comma separated list of the ids.</p>
<p>Iceberg would build the desired reader schema with their schema evolution rules and pass that down to the ORC reader, which would then use its schema evolution to map that to the writer’s schema. Basically, Iceberg would need to change the names of columns and fields to get the desired mapping.</p>
<table>
<thead>
@@ -1340,7 +1340,7 @@ Timestamps <em>without time zone</em> represent a date and time of day regardles
</tr>
<tr>
<td><strong><code>double</code></strong></td>
-<td><code>hashLong(doubleToRawLongBits(v))</code></td>
+<td><code>hashLong(doubleToLongBits(v))</code></td>
<td><code>1.0D</code> ￫ <code>-142385009</code></td>
</tr>
<tr>