You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@parquet.apache.org by ga...@apache.org on 2019/10/29 16:25:32 UTC
[parquet-site] 09/12: PARQUET-1244: Documentation link to logical
types broken
This is an automated email from the ASF dual-hosted git repository.
gabor pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/parquet-site.git
commit 93453c78e402af1c66d359e0c5879a2aa5ebdbb4
Author: Gábor Szádovszky <ga...@apache.org>
AuthorDate: Wed May 30 11:01:59 2018 +0000
PARQUET-1244: Documentation link to logical types broken
Author: Nandor Kollar <nk...@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/parquet/site@1832532 13f79535-47bb-0310-9956-ffa450edef68
---
publish/adopters/index.html | 5 +-
publish/bylaws/index.html | 5 +-
publish/community/index.html | 7 +--
publish/contribute/index.html | 5 +-
publish/developers/index.html | 5 +-
publish/documentation/how-to-release/index.html | 5 +-
publish/documentation/latest/index.html | 74 ++++++++++++++-----------
publish/downloads/index.html | 5 +-
publish/index.html | 5 +-
publish/presentations/index.html | 5 +-
source/_footer.md.erb | 2 +-
source/_header.md.erb | 3 +-
source/community.html.md | 2 +-
source/documentation/latest.html.md | 47 +++++++++-------
14 files changed, 92 insertions(+), 83 deletions(-)
diff --git a/publish/adopters/index.html b/publish/adopters/index.html
index 384efe6..ef6d496 100644
--- a/publish/adopters/index.html
+++ b/publish/adopters/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -158,7 +157,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/bylaws/index.html b/publish/bylaws/index.html
index d2353d6..d297ab1 100644
--- a/publish/bylaws/index.html
+++ b/publish/bylaws/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -132,7 +131,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/community/index.html b/publish/community/index.html
index d6a2e85..a723d6d 100644
--- a/publish/community/index.html
+++ b/publish/community/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -135,7 +134,7 @@
<h4 name="contribute">Contribute a core patch</h4>
<p>Follow our <a href="/contribute/">contribution guidelines</a> when submitting a patch.</p>
<h3>Stay in Touch</h3>
- <p>Attend a Parquet <a href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8">sync up meeting</a> via Google Hangouts.</p>
+ <p>Attend a Parquet sync up meeting via Google Hangouts. These meetings are hold ad-hoc, announced on the developer list about a week before the meeting.</p>
<h3>Bylaws</h3>
<p><a href="/bylaws/">Parquet community bylaws</a></p>
@@ -159,7 +158,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/contribute/index.html b/publish/contribute/index.html
index cae3b8e..84d7083 100644
--- a/publish/contribute/index.html
+++ b/publish/contribute/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -238,7 +237,7 @@ the live site. </p>
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/developers/index.html b/publish/developers/index.html
index d5f0321..4f92130 100644
--- a/publish/developers/index.html
+++ b/publish/developers/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -137,7 +136,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/documentation/how-to-release/index.html b/publish/documentation/how-to-release/index.html
index 62efa0b..029ea3c 100644
--- a/publish/documentation/how-to-release/index.html
+++ b/publish/documentation/how-to-release/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -271,7 +270,7 @@ git push apache :apache-parquet-<VERSION> # delete in the Apache repo
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/documentation/latest/index.html b/publish/documentation/latest/index.html
index 2ef62ec..3efa45c 100644
--- a/publish/documentation/latest/index.html
+++ b/publish/documentation/latest/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -119,7 +118,7 @@
<p>We created Parquet to make the advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem.</p>
-<p>Parquet is built from the ground up with complex nested data structures in mind, and uses the <a href="https://github.com/Parquet/parquet-mr/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper">record shredding and assembly algorithm</a> described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces.</p>
+<p>Parquet is built from the ground up with complex nested data structures in mind, and uses the <a href="https://github.com/julienledem/redelm/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper">record shredding and assembly algorithm</a> described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces.</p>
<p>Parquet is built to support very efficient compression and encoding schemes. Multiple projects have demonstrated the performance impact of applying the right compression and encoding scheme to the data. Parquet allows compression schemes to be specified on a per-column level, and is future-proofed to allow adding more encodings as they are invented and implemented.</p>
@@ -127,11 +126,15 @@
<h2 id="modules">Modules</h2>
-<p>The <code>parquet-format</code> project contains format specifications and Thrift definitions of metadata required to properly read Parquet files.</p>
+<p>The <a href="https://github.com/apache/parquet-format">parquet-format</a> project contains format specifications and Thrift definitions of metadata required to properly read Parquet files.</p>
-<p>The <code>parquet-mr</code> project contains multiple sub-modules, which implement the core components of reading and writing a nested, column-oriented data stream, map this core onto the parquet format, and provide Hadoop Input/Output Formats, Pig loaders, and other java-based utilities for interacting with Parquet.</p>
+<p>The <a href="https://github.com/apache/parquet-mr">parquet-mr</a> project contains multiple sub-modules, which implement the core components of reading and writing a nested, column-oriented data stream, map this core onto the parquet format, and provide Hadoop Input/Output Formats, Pig loaders, and other Java-based utilities for interacting with Parquet.</p>
-<p>The <code>parquet-compatibility</code> project contains compatibility tests that can be used to verify that implementations in different languages can read and write each other’s files.</p>
+<p>The <a href="https://github.com/apache/parquet-cpp">parquet-cpp</a> project is a C++ library to read-write Parquet files.</p>
+
+<p>The <a href="https://github.com/sunchao/parquet-rs">parquet-rs</a> project is a Rust library to read-write Parquet files.</p>
+
+<p>The <a href="https://github.com/Parquet/parquet-compatibility">parquet-compatibility</a> project contains compatibility tests that can be used to verify that implementations in different languages can read and write each other’s files.</p>
<h2 id="building">Building</h2>
@@ -206,29 +209,32 @@ in the thrift files.</p>
<p>Readers are expected to first read the file metadata to find all the column
chunks they are interested in. The columns chunks should then be read sequentially.</p>
-<p><img alt="File Layout" src="https://raw.github.com/Parquet/parquet-format/master/doc/images/FileLayout.gif" /></p>
+<p><img alt="File Layout" src="https://raw.github.com/apache/parquet-format/master/doc/images/FileLayout.gif" /></p>
<h2 id="metadata">Metadata</h2>
<p>There are three types of metadata: file metadata, column (chunk) metadata and page
header metadata. All thrift structures are serialized using the TCompactProtocol.</p>
-<p><img alt="Metadata diagram" src="https://github.com/Parquet/parquet-format/raw/master/doc/images/FileFormat.gif" /></p>
+<p><img alt="Metadata diagram" src="https://github.com/apache/parquet-format/raw/master/doc/images/FileFormat.gif" /></p>
<h2 id="types">Types</h2>
<p>The types supported by the file format are intended to be as minimal as possible,
-with a focus on how the types effect on disk storage. For example, 16-bit ints
+with a focus on how the types effect on disk storage. For example, 16-bit ints
are not explicitly supported in the storage format since they are covered by
-32-bit ints with an efficient encoding. This reduces the complexity of implementing
-readers and writers for the format. The types are:
- - BOOLEAN: 1 bit boolean
- - INT32: 32 bit signed ints
- - INT64: 64 bit signed ints
- - INT96: 96 bit signed ints
- - FLOAT: IEEE 32-bit floating point values
- - DOUBLE: IEEE 64-bit floating point values
- - BYTE_ARRAY: arbitrarily long byte arrays.</p>
+32-bit ints with an efficient encoding. This reduces the complexity of implementing
+readers and writers for the format. The types are:</p>
+
+<ul>
+<li><em>BOOLEAN</em>: 1 bit boolean</li>
+<li><em>INT32</em>: 32 bit signed ints</li>
+<li><em>INT64</em>: 64 bit signed ints</li>
+<li><em>INT96</em>: 96 bit signed ints</li>
+<li><em>FLOAT</em>: IEEE 32-bit floating point values</li>
+<li><em>DOUBLE</em>: IEEE 64-bit floating point values</li>
+<li><em>BYTE_ARRAY</em>: arbitrarily long byte arrays.</li>
+</ul>
<h3 id="logical-types">Logical Types</h3>
@@ -239,7 +245,7 @@ example, strings are stored as byte arrays (binary) with a UTF8 annotation.
These annotations define how to further decode and interpret the data.
Annotations are stored as a <code>ConvertedType</code> in the file metadata and are
documented in
-<a href="https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md">LogicalTypes.md</a>.</p>
+<a href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md">LogicalTypes.md</a>.</p>
<h2 id="nested-encoding">Nested Encoding</h2>
@@ -263,11 +269,14 @@ nothing else. </p>
<h2 id="data-pages">Data Pages</h2>
<p>For data pages, the 3 pieces of information are encoded back to back, after the page
-header. We have the
- - definition levels data,<br>
- - repetition levels data,
- - encoded values.
-The size of specified in the header is for all 3 pieces combined.</p>
+header. We have the </p>
+
+<ul>
+<li>definition levels data,<br></li>
+<li>repetition levels data, </li>
+<li>encoded values.
+The size of specified in the header is for all 3 pieces combined.</li>
+</ul>
<p>The data for the data page is always required. The definition and repetition levels
are optional, based on the schema definition. If the column is not nested (i.e.
@@ -278,7 +287,7 @@ skipped (if encoded, it will always have the value of the max definition level).
<p>For example, in the case where the column is non-nested and required, the data in the
page is only the encoded values.</p>
-<p>The supported encodings are described in <a href="https://github.com/Parquet/parquet-format/blob/master/Encodings.md">Encodings.md</a></p>
+<p>The supported encodings are described in <a href="https://github.com/apache/parquet-format/blob/master/Encodings.md">Encodings.md</a></p>
<h2 id="column-chunks">Column chunks</h2>
@@ -310,7 +319,7 @@ a reader could recover partially written files. </p>
<h2 id="separating-metadata-and-column-data.">Separating metadata and column data.</h2>
-<p>The format is explicitly designed to separate the metadata from the data. This
+<p>The format is explicitly designed to separate the metadata from the data. This
allows splitting columns into multiple files, as well as having a single metadata
file reference multiple parquet files. </p>
@@ -333,10 +342,13 @@ at a time; this is not the IO chunk. We recommend 8KB for page sizes.</li>
<h2 id="extensibility">Extensibility</h2>
-<p>There are many places in the format for compatible extensions:
-- File Version: The file metadata contains a version.
-- Encodings: Encodings are specified by enum and more can be added in the future.<br>
-- Page types: Additional page types can be added and safely skipped.</p>
+<p>There are many places in the format for compatible extensions:</p>
+
+<ul>
+<li>File Version: The file metadata contains a version.</li>
+<li>Encodings: Encodings are specified by enum and more can be added in the future.<br></li>
+<li>Page types: Additional page types can be added and safely skipped.</li>
+</ul>
</div>
<div class="container">
@@ -345,7 +357,7 @@ at a time; this is not the IO chunk. We recommend 8KB for page sizes.</li>
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/downloads/index.html b/publish/downloads/index.html
index 48a076a..118fc73 100644
--- a/publish/downloads/index.html
+++ b/publish/downloads/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -146,7 +145,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/index.html b/publish/index.html
index e269302..e5db0a8 100644
--- a/publish/index.html
+++ b/publish/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -148,7 +147,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/publish/presentations/index.html b/publish/presentations/index.html
index 98b16df..e936947 100644
--- a/publish/presentations/index.html
+++ b/publish/presentations/index.html
@@ -51,12 +51,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
@@ -144,7 +143,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/source/_footer.md.erb b/source/_footer.md.erb
index 641d8c2..b8f6f85 100644
--- a/source/_footer.md.erb
+++ b/source/_footer.md.erb
@@ -4,7 +4,7 @@
<div class="row-fluid">
<div class="span12 text-left">
<div class="span12">
- Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
+ Copyright 2018 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. Apache Parquet and the Apache feather logo are trademarks of The Apache Software Foundation.
</div>
</div>
diff --git a/source/_header.md.erb b/source/_header.md.erb
index 7f7ccad..29c1558 100644
--- a/source/_header.md.erb
+++ b/source/_header.md.erb
@@ -21,12 +21,11 @@
<a href="/community">Get Involved <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu" aria-labelledby="drop1">
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://issues.apache.org/jira/browse/parquet"><i class="fa fa-bug"></i> JIRA (Bugs)</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://git-wip-us.apache.org/repos/asf?s=parquet"><i class="fa fa-code"></i> Source (Apache)</a></li>
+ <li role="presentation"><a role="menuitem" tabindex="-1" href="https://gitbox.apache.org/repos/asf?p=parquet-mr.git"><i class="fa fa-code"></i> Source (Apache)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://github.com/apache/parquet-mr"><i class="fa fa-github-alt"></i> Source (GitHub)</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="/contribute"><i class="fa fa-code-fork"></i> Contributing</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="https://twitter.com/ApacheParquet"><i class="fa fa-twitter"></i> @ApacheParquet</a></li>
<li role="presentation"><a role="menuitem" tabindex="-1" href="http://stackoverflow.com/questions/tagged/parquet"><i class="fa fa-stack-overflow"></i> StackOverflow</a></li>
- <li role="presentation"><a role="menuitem" tabindex="-1" href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8"><i class="fa fa-google"></i> Google Hangout</a></li>
</ul>
</li>
</ul>
diff --git a/source/community.html.md b/source/community.html.md
index 720bf92..3c808f1 100644
--- a/source/community.html.md
+++ b/source/community.html.md
@@ -17,7 +17,7 @@
<h4 name="contribute">Contribute a core patch</h4>
<p>Follow our <a href="/contribute/">contribution guidelines</a> when submitting a patch.</p>
<h3>Stay in Touch</h3>
- <p>Attend a Parquet <a href="https://plus.google.com/events/c36apc97f7invko9p128hq9e6b8">sync up meeting</a> via Google Hangouts.</p>
+ <p>Attend a Parquet sync up meeting via Google Hangouts. These meetings are hold ad-hoc, announced on the developer list about a week before the meeting.</p>
<h3>Bylaws</h3>
<p><a href="/bylaws/">Parquet community bylaws</a></p>
diff --git a/source/documentation/latest.html.md b/source/documentation/latest.html.md
index 4d5d1a1..ad3a6b6 100644
--- a/source/documentation/latest.html.md
+++ b/source/documentation/latest.html.md
@@ -2,7 +2,7 @@
We created Parquet to make the advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem.
-Parquet is built from the ground up with complex nested data structures in mind, and uses the [record shredding and assembly algorithm](https://github.com/Parquet/parquet-mr/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper) described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces.
+Parquet is built from the ground up with complex nested data structures in mind, and uses the [record shredding and assembly algorithm](https://github.com/julienledem/redelm/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper) described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces.
Parquet is built to support very efficient compression and encoding schemes. Multiple projects have demonstrated the performance impact of applying the right compression and encoding scheme to the data. Parquet allows compression schemes to be specified on a per-column level, and is future-proofed to allow adding more encodings as they are invented and implemented.
@@ -10,11 +10,15 @@ Parquet is built to be used by anyone. The Hadoop ecosystem is rich with data pr
## Modules
-The `parquet-format` project contains format specifications and Thrift definitions of metadata required to properly read Parquet files.
+The [parquet-format](https://github.com/apache/parquet-format) project contains format specifications and Thrift definitions of metadata required to properly read Parquet files.
-The `parquet-mr` project contains multiple sub-modules, which implement the core components of reading and writing a nested, column-oriented data stream, map this core onto the parquet format, and provide Hadoop Input/Output Formats, Pig loaders, and other java-based utilities for interacting with Parquet.
+The [parquet-mr](https://github.com/apache/parquet-mr) project contains multiple sub-modules, which implement the core components of reading and writing a nested, column-oriented data stream, map this core onto the parquet format, and provide Hadoop Input/Output Formats, Pig loaders, and other Java-based utilities for interacting with Parquet.
-The `parquet-compatibility` project contains compatibility tests that can be used to verify that implementations in different languages can read and write each other's files.
+The [parquet-cpp](https://github.com/apache/parquet-cpp) project is a C++ library to read-write Parquet files.
+
+The [parquet-rs](https://github.com/sunchao/parquet-rs) project is a Rust library to read-write Parquet files.
+
+The [parquet-compatibility](https://github.com/Parquet/parquet-compatibility) project contains compatibility tests that can be used to verify that implementations in different languages can read and write each other's files.
## Building
@@ -89,27 +93,28 @@ Metadata is written after the data to allow for single pass writing.
Readers are expected to first read the file metadata to find all the column
chunks they are interested in. The columns chunks should then be read sequentially.
- 
+ 
## Metadata
There are three types of metadata: file metadata, column (chunk) metadata and page
header metadata. All thrift structures are serialized using the TCompactProtocol.
- 
+ 
## Types
The types supported by the file format are intended to be as minimal as possible,
-with a focus on how the types effect on disk storage. For example, 16-bit ints
+with a focus on how the types effect on disk storage. For example, 16-bit ints
are not explicitly supported in the storage format since they are covered by
-32-bit ints with an efficient encoding. This reduces the complexity of implementing
-readers and writers for the format. The types are:
- - BOOLEAN: 1 bit boolean
- - INT32: 32 bit signed ints
- - INT64: 64 bit signed ints
- - INT96: 96 bit signed ints
- - FLOAT: IEEE 32-bit floating point values
- - DOUBLE: IEEE 64-bit floating point values
- - BYTE_ARRAY: arbitrarily long byte arrays.
+32-bit ints with an efficient encoding. This reduces the complexity of implementing
+readers and writers for the format. The types are:
+
+ - *BOOLEAN*: 1 bit boolean
+ - *INT32*: 32 bit signed ints
+ - *INT64*: 64 bit signed ints
+ - *INT96*: 96 bit signed ints
+ - *FLOAT*: IEEE 32-bit floating point values
+ - *DOUBLE*: IEEE 64-bit floating point values
+ - *BYTE_ARRAY*: arbitrarily long byte arrays.
### Logical Types
Logical types are used to extend the types that parquet can be used to store,
@@ -121,7 +126,7 @@ Annotations are stored as a `ConvertedType` in the file metadata and are
documented in
[LogicalTypes.md][logical-types].
-[logical-types]: https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md
+[logical-types]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
## Nested Encoding
To encode nested columns, Parquet uses the Dremel encoding with definition and
@@ -142,7 +147,8 @@ nothing else.
## Data Pages
For data pages, the 3 pieces of information are encoded back to back, after the page
-header. We have the
+header. We have the
+
- definition levels data,
- repetition levels data,
- encoded values.
@@ -157,7 +163,7 @@ skipped (if encoded, it will always have the value of the max definition level).
For example, in the case where the column is non-nested and required, the data in the
page is only the encoded values.
-The supported encodings are described in [Encodings.md](https://github.com/Parquet/parquet-format/blob/master/Encodings.md)
+The supported encodings are described in [Encodings.md](https://github.com/apache/parquet-format/blob/master/Encodings.md)
## Column chunks
Column chunks are composed of pages written back to back. The pages share a common
@@ -185,7 +191,7 @@ far. Combining this with the strategy used for rc or avro files using sync mark
a reader could recover partially written files.
## Separating metadata and column data.
-The format is explicitly designed to separate the metadata from the data. This
+The format is explicitly designed to separate the metadata from the data. This
allows splitting columns into multiple files, as well as having a single metadata
file reference multiple parquet files.
@@ -205,6 +211,7 @@ at a time; this is not the IO chunk. We recommend 8KB for page sizes.
## Extensibility
There are many places in the format for compatible extensions:
+
- File Version: The file metadata contains a version.
- Encodings: Encodings are specified by enum and more can be added in the future.
- Page types: Additional page types can be added and safely skipped.