You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@orc.apache.org by om...@apache.org on 2019/05/03 00:05:54 UTC
[orc] branch asf-site updated: Push ORC-483 changes to site.
This is an automated email from the ASF dual-hosted git repository.
omalley pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/orc.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ecae50a Push ORC-483 changes to site.
ecae50a is described below
commit ecae50ac9b071d9ed33d0048bf3d316573d20625
Author: Owen O'Malley <om...@apache.org>
AuthorDate: Thu May 2 17:05:20 2019 -0700
Push ORC-483 changes to site.
Signed-off-by: Owen O'Malley <om...@apache.org>
---
docs/adopters.html | 2 +-
specification/ORCv1/index.html | 314 ++++++++++++++++++++++++++++++++++++++---
specification/ORCv2/index.html | 314 ++++++++++++++++++++++++++++++++++++++---
3 files changed, 589 insertions(+), 41 deletions(-)
diff --git a/docs/adopters.html b/docs/adopters.html
index 6a745db..8b93b2e 100644
--- a/docs/adopters.html
+++ b/docs/adopters.html
@@ -728,7 +728,7 @@ or directly into Hive tables backed by an ORC file format.</p>
<p>With more than 300 PB of data, Facebook was an <a href="https://code.facebook.com/posts/229861827208629/scaling-the-facebook-data-warehouse-to-300-pb/">early adopter of
ORC</a> and quickly put it into production.</p>
-<h3 id="presto"><a href="https://prestodb.io/">Presto</a></h3>
+<h3 id="presto"><a href="https://prestosql.io/">Presto</a></h3>
<p>The Presto team has done a lot of work <a href="https://code.facebook.com/posts/370832626374903/even-faster-data-at-the-speed-of-presto-orc/">integrating
ORC</a> into their SQL engine.</p>
diff --git a/specification/ORCv1/index.html b/specification/ORCv1/index.html
index 6b57eb2..54c62c0 100644
--- a/specification/ORCv1/index.html
+++ b/specification/ORCv1/index.html
@@ -126,6 +126,15 @@ incorporates the Protobuf definition from the
reader is encouraged to review the Protobuf encoding if they need to
understand the byte-level encoding</p>
+<p>The sections of the file tail are (and their protobuf message type):</p>
+<ul>
+ <li>encrypted stripe statistics: list of ColumnarStripeStatistics</li>
+ <li>stripe statistics: Metadata</li>
+ <li>footer: Footer</li>
+ <li>postscript: PostScript</li>
+ <li>psLen: byte</li>
+</ul>
+
<h2 id="postscript">Postscript</h2>
<p>The Postscript section provides the necessary information to interpret
@@ -201,6 +210,16 @@ information as described in this section.</p>
repeated ColumnStatistics statistics = 7;
// the maximum number of rows in each index entry
optional uint32 rowIndexStride = 8;
+ // Each implementation that writes ORC files should register for a code
+ // 0 = ORC Java
+ // 1 = ORC C++
+ // 2 = Presto
+ // 3 = Scritchley Go from https://github.com/scritchley/orc
+ optional uint32 writer = 9;
+ // information about the encryption in this file
+ optional Encryption encryption = 10;
+ // the number of bytes in the encrypted stripe statistics
+ optional uint64 stripeStatisticsLength = 11;
}
</code></pre></div></div>
@@ -215,6 +234,17 @@ itself, and a stripe footer. Both the indexes and the data sections
are divided by columns so that only the data for the required columns
needs to be read.</p>
+<p>The encryptStripeId and encryptedLocalKeys support column
+encryption. They are set on the first stripe of each ORC file with
+column encryption and not set after that. For a stripe with the values
+set, the reader should use those values for that stripe. Subsequent
+stripes use the previous encryptStripeId + 1 and the same keys.</p>
+
+<p>The current ORC merging code merges entire files, and thus the reader
+will get the correct values on what was the first stripe and continue
+on. If we develop a merge tool that reorders stripes or does partial
+merges, these values will need to be set correctly by that tool.</p>
+
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message StripeInformation {
// the start of the stripe within the file
optional uint64 offset = 1;
@@ -226,6 +256,19 @@ needs to be read.</p>
optional uint64 footerLength = 4;
// the number of rows in the stripe
optional uint64 numberOfRows = 5;
+ // If this is present, the reader should use this value for the encryption
+ // stripe id for setting the encryption IV. Otherwise, the reader should
+ // use one larger than the previous stripe's encryptStripeId.
+ // For unmerged ORC files, the first stripe will use 1 and the rest of the
+ // stripes won't have it set. For merged files, the stripe information
+ // will be copied from their original files and thus the first stripe of
+ // each of the input files will reset it to 1.
+ // Note that 1 was choosen, because protobuf v3 doesn't serialize
+ // primitive types that are the default (eg. 0).
+ optional uint64 encryptStripeId = 6;
+ // For each encryption variant, the new encrypted local key to use until we
+ // find a replacement.
+ repeated bytes encryptedLocalKeys = 7;
}
</code></pre></div></div>
@@ -432,6 +475,195 @@ based on the predicate push-down evaluated per a stripe.</p>
}
</code></pre></div></div>
+<h1 id="column-encryption">Column Encryption</h1>
+
+<p>ORC as of Apache ORC 1.6 supports column encryption where the data and
+statistics of specific columns are encrypted on disk. Column
+encryption provides fine-grain column level security even when many
+users have access to the file itself. The encryption is transparent to
+the user and the writer only needs to define which columns and
+encryption keys to use. When reading an ORC file, if the user has
+access to the keys, they will get the real data. If they do not have
+the keys, they will get the masked data.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message Encryption {
+ // all of the masks used in this file
+ repeated DataMask mask = 1;
+ // all of the keys used in this file
+ repeated EncryptionKey key = 2;
+ // The encrypted variants.
+ // Readers should prefer the first variant that the user has access to
+ // the corresponding key. If they don't have access to any of the keys,
+ // they should get the unencrypted masked data.
+ repeated EncryptionVariant variants = 3;
+ // How are the local keys encrypted?
+ optional KeyProviderKind keyProvider = 4;
+}
+</code></pre></div></div>
+
+<p>Each encrypted column in each file will have a random local key
+generated for it. Thus, even though all of the decryption happens
+locally in the reader, a malicious user that stores the key only
+enables access that column in that file. The local keys are encrypted
+by the Hadoop or Ranger Key Management Server (KMS). The encrypted
+local keys are stored in the file footer’s StripeInformation.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enum KeyProviderKind {
+ UNKNOWN = 0;
+ HADOOP = 1;
+ AWS = 2;
+ GCP = 3;
+ AZURE = 4;
+}
+</code></pre></div></div>
+
+<p>When ORC is using the Hadoop or Ranger KMS, it generates a random encrypted
+local key (16 or 32 bytes for 128 or 256 bit AES respectively). Using the
+first 16 bytes as the IV, it uses AES/CTR to decrypt the local key.</p>
+
+<p>With the AWS KMS, the GenerateDataKey method is used to create a new local
+key and the Decrypt method is used to decrypt it.</p>
+
+<h2 id="data-masks">Data Masks</h2>
+
+<p>The user’s data is statically masked before writing the unencrypted
+variant. Because the masking was done statically when the file was
+written, the information about the masking is just informational.</p>
+
+<p>The three standard masks are:</p>
+
+<ul>
+ <li>nullify - all values become null</li>
+ <li>redact - replace characters with constants such as X or 9</li>
+ <li>sha256 - replace string with the SHA 256 of the value</li>
+</ul>
+
+<p>The default is nullify, but masks may be defined by the user. Masks
+are not allowed to change the type of the column, just the values.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message DataMask {
+ // the kind of masking, which may include third party masks
+ optional string name = 1;
+ // parameters for the mask
+ repeated string maskParameters = 2;
+ // the unencrypted column roots this mask was applied to
+ repeated uint32 columns = 3 [packed = true];
+}
+</code></pre></div></div>
+
+<h2 id="encryption-keys">Encryption Keys</h2>
+
+<p>In addition to the encrypted local keys, which are stored in the
+footer’s StripeInformation, the file also needs to describe the master
+key that was used to encrypt the local keys. The master keys are
+described by name, their version, and the encryption algorithm.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message EncryptionKey {
+ optional string keyName = 1;
+ optional uint32 keyVersion = 2;
+ optional EncryptionAlgorithm algorithm = 3;
+}
+</code></pre></div></div>
+
+<p>The encryption algorithm is stored using an enumeration and since
+ProtoBuf uses the 0 value as a default, we added an unused value. That
+ensures that if we add a new algorithm that old readers will get
+UNKNOWN_ENCRYPTION instead of a real value.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enum EncryptionAlgorithm {
+ // used for detecting future algorithms
+ UNKNOWN_ENCRYPTION = 0;
+ // 128 bit AES/CTR
+ AES_CTR_128 = 1;
+ // 256 bit AES/CTR
+ AES_CTR_256 = 2;
+}
+</code></pre></div></div>
+
+<h2 id="encryption-variants">Encryption Variants</h2>
+
+<p>Each encrypted column is written as two variants:</p>
+
+<ul>
+ <li>encrypted unmasked - for users with access to the key</li>
+ <li>unencrypted masked - for all other users</li>
+</ul>
+
+<p>The changes to the format were done so that old ORC readers will read
+the masked unencrypted data. Encryption variants encrypt a subtree of
+columns and use a single local key. The initial version of encryption
+support only allows the two variants, but this may be extended later
+and thus readers should use the first variant of a column that the
+reader has access to.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message EncryptionVariant {
+ // the column id of the root column that is encrypted in this variant
+ optional uint32 root = 1;
+ // the key that encrypted this variant
+ optional uint32 key = 2;
+ // The master key that was used to encrypt the local key, referenced as
+ // an index into the Encryption.key list.
+ optional bytes encryptedKey = 3;
+ // the stripe statistics for this variant
+ repeated Stream stripeStatistics = 4;
+ // encrypted file statistics as a FileStatistics
+ optional bytes fileStatistics = 5;
+}
+</code></pre></div></div>
+
+<p>Each variant stores stripe and file statistics separately. The file
+statistics are serialized as a FileStatistics, compressed, encrypted
+and stored in the EncryptionVariant.fileStatistics.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message FileStatistics {
+ repeated ColumnStatistics column = 1;
+}
+</code></pre></div></div>
+
+<p>The stripe statistics for each column are serialized as
+ColumnarStripeStatistics, compressed, encrypted and stored in a stream
+of kind STRIPE_STATISTICS. By making the column stripe statistics
+independent of each other, the reader only reads and parses the
+columns contained in the SARG.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message ColumnarStripeStatistics {
+ // one value for each stripe in the file
+ repeated ColumnStatistics colStats = 1;
+}
+</code></pre></div></div>
+
+<h2 id="stream-encryption">Stream Encryption</h2>
+
+<p>Our encryption is done using AES/CTR. CTR is a mode that has some very
+nice properties for us:</p>
+
+<ul>
+ <li>It is seeded so that identical data is encrypted differently.</li>
+ <li>It does not require padding the stream to the cipher length.</li>
+ <li>It allows readers to seek in to a stream.</li>
+ <li>The IV does not need to be randomly generated.</li>
+</ul>
+
+<p>To ensure that we don’t reuse IV, we set the IV as:</p>
+
+<ul>
+ <li>bytes 0 to 2 - column id</li>
+ <li>bytes 3 to 4 - stream kind</li>
+ <li>bytes 5 to 7 - stripe id</li>
+ <li>bytes 8 to 15 - cipher block counter</li>
+</ul>
+
+<p>However, it is critical for CTR that we never reuse an initialization
+vector (IV) with the same local key.</p>
+
+<p>For data in the footer, use the number of stripes in the file as the
+stripe id. This guarantees when we write an intermediate footer in to
+a file that we don’t use the same IV.</p>
+
+<p>Additionally, we never reuse a local key for new data. For example, when
+merging files, we don’t reuse local key from the input files for the new
+file tail, but always generate a new local key.</p>
+
<h1 id="compression">Compression</h1>
<p>If the ORC file writer selects a generic compression codec (zlib or
@@ -879,6 +1111,23 @@ uses three streams PRESENT, DATA, and LENGTH, which stores the length
of each value. The details of each type will be presented in the
following subsections.</p>
+<p>The layout of each stripe looks like:</p>
+<ul>
+ <li>index streams
+ <ul>
+ <li>unencrypted</li>
+ <li>encryption variant 1..N</li>
+ </ul>
+ </li>
+ <li>data streams
+ <ul>
+ <li>unencrypted</li>
+ <li>encryption variant 1..N</li>
+ </ul>
+ </li>
+ <li>stripe footer</li>
+</ul>
+
<h2 id="stripe-footer">Stripe Footer</h2>
<p>The stripe footer contains the encoding of each column and the
@@ -889,6 +1138,22 @@ directory of the streams including their location.</p>
repeated Stream streams = 1;
// the encoding of each column
repeated ColumnEncoding columns = 2;
+ optional string writerTimezone = 3;
+ // one for each column encryption variant
+ repeated StripeEncryptionVariant encryption = 4;
+}
+</code></pre></div></div>
+
+<p>If the file includes encrypted columns, those streams and column
+encodings are stored separately in a StripeEncryptionVariant per an
+encryption variant. Additionally, the StripeFooter will contain two
+additional virtual streams ENCRYPTED_INDEX and ENCRYPTED_DATA that
+allocate the space that is used by the encryption variants to store
+the encrypted index and data streams.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message StripeEncryptionVariant {
+ repeated Stream streams = 1;
+ repeated ColumnEncoding encoding = 2;
}
</code></pre></div></div>
@@ -898,26 +1163,35 @@ depends on the type and encoding of the column.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message Stream {
enum Kind {
- // boolean stream of whether the next value is non-null
- PRESENT = 0;
- // the primary data stream
- DATA = 1;
- // the length of each value for variable length data
- LENGTH = 2;
- // the dictionary blob
- DICTIONARY_DATA = 3;
- // deprecated prior to Hive 0.11
- // It was used to store the number of instances of each value in the
- // dictionary
- DICTIONARY_COUNT = 4;
- // a secondary data stream
- SECONDARY = 5;
- // the index for seeking to particular row groups
- ROW_INDEX = 6;
- // original bloom filters used before ORC-101
- BLOOM_FILTER = 7;
- // bloom filters that consistently use utf8
- BLOOM_FILTER_UTF8 = 8;
+ // boolean stream of whether the next value is non-null
+ PRESENT = 0;
+ // the primary data stream
+ DATA = 1;
+ // the length of each value for variable length data
+ LENGTH = 2;
+ // the dictionary blob
+ DICTIONARY_DATA = 3;
+ // deprecated prior to Hive 0.11
+ // It was used to store the number of instances of each value in the
+ // dictionary
+ DICTIONARY_COUNT = 4;
+ // a secondary data stream
+ SECONDARY = 5;
+ // the index for seeking to particular row groups
+ ROW_INDEX = 6;
+ // original bloom filters used before ORC-101
+ BLOOM_FILTER = 7;
+ // bloom filters that consistently use utf8
+ BLOOM_FILTER_UTF8 = 8;
+
+ // Virtual stream kinds to allocate space for encrypted index and data.
+ ENCRYPTED_INDEX = 9;
+ ENCRYPTED_DATA = 10;
+
+ // stripe statistics streams
+ STRIPE_STATISTICS = 100;
+ // A virtual stream kind that is used for setting the encryption IV.
+ FILE_STATISTICS = 101;
}
required Kind kind = 1;
// the column id
diff --git a/specification/ORCv2/index.html b/specification/ORCv2/index.html
index c348a25..44f3b0f 100644
--- a/specification/ORCv2/index.html
+++ b/specification/ORCv2/index.html
@@ -151,6 +151,15 @@ incorporates the Protobuf definition from the
reader is encouraged to review the Protobuf encoding if they need to
understand the byte-level encoding</p>
+<p>The sections of the file tail are (and their protobuf message type):</p>
+<ul>
+ <li>encrypted stripe statistics: list of ColumnarStripeStatistics</li>
+ <li>stripe statistics: Metadata</li>
+ <li>footer: Footer</li>
+ <li>postscript: PostScript</li>
+ <li>psLen: byte</li>
+</ul>
+
<h2 id="postscript">Postscript</h2>
<p>The Postscript section provides the necessary information to interpret
@@ -226,6 +235,16 @@ information as described in this section.</p>
repeated ColumnStatistics statistics = 7;
// the maximum number of rows in each index entry
optional uint32 rowIndexStride = 8;
+ // Each implementation that writes ORC files should register for a code
+ // 0 = ORC Java
+ // 1 = ORC C++
+ // 2 = Presto
+ // 3 = Scritchley Go from https://github.com/scritchley/orc
+ optional uint32 writer = 9;
+ // information about the encryption in this file
+ optional Encryption encryption = 10;
+ // the number of bytes in the encrypted stripe statistics
+ optional uint64 stripeStatisticsLength = 11;
}
</code></pre></div></div>
@@ -240,6 +259,17 @@ itself, and a stripe footer. Both the indexes and the data sections
are divided by columns so that only the data for the required columns
needs to be read.</p>
+<p>The encryptStripeId and encryptedLocalKeys support column
+encryption. They are set on the first stripe of each ORC file with
+column encryption and not set after that. For a stripe with the values
+set, the reader should use those values for that stripe. Subsequent
+stripes use the previous encryptStripeId + 1 and the same keys.</p>
+
+<p>The current ORC merging code merges entire files, and thus the reader
+will get the correct values on what was the first stripe and continue
+on. If we develop a merge tool that reorders stripes or does partial
+merges, these values will need to be set correctly by that tool.</p>
+
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message StripeInformation {
// the start of the stripe within the file
optional uint64 offset = 1;
@@ -251,6 +281,19 @@ needs to be read.</p>
optional uint64 footerLength = 4;
// the number of rows in the stripe
optional uint64 numberOfRows = 5;
+ // If this is present, the reader should use this value for the encryption
+ // stripe id for setting the encryption IV. Otherwise, the reader should
+ // use one larger than the previous stripe's encryptStripeId.
+ // For unmerged ORC files, the first stripe will use 1 and the rest of the
+ // stripes won't have it set. For merged files, the stripe information
+ // will be copied from their original files and thus the first stripe of
+ // each of the input files will reset it to 1.
+ // Note that 1 was choosen, because protobuf v3 doesn't serialize
+ // primitive types that are the default (eg. 0).
+ optional uint64 encryptStripeId = 6;
+ // For each encryption variant, the new encrypted local key to use until we
+ // find a replacement.
+ repeated bytes encryptedLocalKeys = 7;
}
</code></pre></div></div>
@@ -456,6 +499,195 @@ based on the predicate push-down evaluated per a stripe.</p>
}
</code></pre></div></div>
+<h1 id="column-encryption">Column Encryption</h1>
+
+<p>ORC as of Apache ORC 1.6 supports column encryption where the data and
+statistics of specific columns are encrypted on disk. Column
+encryption provides fine-grain column level security even when many
+users have access to the file itself. The encryption is transparent to
+the user and the writer only needs to define which columns and
+encryption keys to use. When reading an ORC file, if the user has
+access to the keys, they will get the real data. If they do not have
+the keys, they will get the masked data.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message Encryption {
+ // all of the masks used in this file
+ repeated DataMask mask = 1;
+ // all of the keys used in this file
+ repeated EncryptionKey key = 2;
+ // The encrypted variants.
+ // Readers should prefer the first variant that the user has access to
+ // the corresponding key. If they don't have access to any of the keys,
+ // they should get the unencrypted masked data.
+ repeated EncryptionVariant variants = 3;
+ // How are the local keys encrypted?
+ optional KeyProviderKind keyProvider = 4;
+}
+</code></pre></div></div>
+
+<p>Each encrypted column in each file will have a random local key
+generated for it. Thus, even though all of the decryption happens
+locally in the reader, a malicious user that stores the key only
+enables access that column in that file. The local keys are encrypted
+by the Hadoop or Ranger Key Management Server (KMS). The encrypted
+local keys are stored in the file footer’s StripeInformation.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enum KeyProviderKind {
+ UNKNOWN = 0;
+ HADOOP = 1;
+ AWS = 2;
+ GCP = 3;
+ AZURE = 4;
+}
+</code></pre></div></div>
+
+<p>When ORC is using the Hadoop or Ranger KMS, it generates a random encrypted
+local key (16 or 32 bytes for 128 or 256 bit AES respectively). Using the
+first 16 bytes as the IV, it uses AES/CTR to decrypt the local key.</p>
+
+<p>With the AWS KMS, the GenerateDataKey method is used to create a new local
+key and the Decrypt method is used to decrypt it.</p>
+
+<h2 id="data-masks">Data Masks</h2>
+
+<p>The user’s data is statically masked before writing the unencrypted
+variant. Because the masking was done statically when the file was
+written, the information about the masking is just informational.</p>
+
+<p>The three standard masks are:</p>
+
+<ul>
+ <li>nullify - all values become null</li>
+ <li>redact - replace characters with constants such as X or 9</li>
+ <li>sha256 - replace string with the SHA 256 of the value</li>
+</ul>
+
+<p>The default is nullify, but masks may be defined by the user. Masks
+are not allowed to change the type of the column, just the values.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message DataMask {
+ // the kind of masking, which may include third party masks
+ optional string name = 1;
+ // parameters for the mask
+ repeated string maskParameters = 2;
+ // the unencrypted column roots this mask was applied to
+ repeated uint32 columns = 3 [packed = true];
+}
+</code></pre></div></div>
+
+<h2 id="encryption-keys">Encryption Keys</h2>
+
+<p>In addition to the encrypted local keys, which are stored in the
+footer’s StripeInformation, the file also needs to describe the master
+key that was used to encrypt the local keys. The master keys are
+described by name, their version, and the encryption algorithm.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message EncryptionKey {
+ optional string keyName = 1;
+ optional uint32 keyVersion = 2;
+ optional EncryptionAlgorithm algorithm = 3;
+}
+</code></pre></div></div>
+
+<p>The encryption algorithm is stored using an enumeration and since
+ProtoBuf uses the 0 value as a default, we added an unused value. That
+ensures that if we add a new algorithm that old readers will get
+UNKNOWN_ENCRYPTION instead of a real value.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enum EncryptionAlgorithm {
+ // used for detecting future algorithms
+ UNKNOWN_ENCRYPTION = 0;
+ // 128 bit AES/CTR
+ AES_CTR_128 = 1;
+ // 256 bit AES/CTR
+ AES_CTR_256 = 2;
+}
+</code></pre></div></div>
+
+<h2 id="encryption-variants">Encryption Variants</h2>
+
+<p>Each encrypted column is written as two variants:</p>
+
+<ul>
+ <li>encrypted unmasked - for users with access to the key</li>
+ <li>unencrypted masked - for all other users</li>
+</ul>
+
+<p>The changes to the format were done so that old ORC readers will read
+the masked unencrypted data. Encryption variants encrypt a subtree of
+columns and use a single local key. The initial version of encryption
+support only allows the two variants, but this may be extended later
+and thus readers should use the first variant of a column that the
+reader has access to.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message EncryptionVariant {
+ // the column id of the root column that is encrypted in this variant
+ optional uint32 root = 1;
+ // the key that encrypted this variant
+ optional uint32 key = 2;
+ // The master key that was used to encrypt the local key, referenced as
+ // an index into the Encryption.key list.
+ optional bytes encryptedKey = 3;
+ // the stripe statistics for this variant
+ repeated Stream stripeStatistics = 4;
+ // encrypted file statistics as a FileStatistics
+ optional bytes fileStatistics = 5;
+}
+</code></pre></div></div>
+
+<p>Each variant stores stripe and file statistics separately. The file
+statistics are serialized as a FileStatistics, compressed, encrypted
+and stored in the EncryptionVariant.fileStatistics.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message FileStatistics {
+ repeated ColumnStatistics column = 1;
+}
+</code></pre></div></div>
+
+<p>The stripe statistics for each column are serialized as
+ColumnarStripeStatistics, compressed, encrypted and stored in a stream
+of kind STRIPE_STATISTICS. By making the column stripe statistics
+independent of each other, the reader only reads and parses the
+columns contained in the SARG.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message ColumnarStripeStatistics {
+ // one value for each stripe in the file
+ repeated ColumnStatistics colStats = 1;
+}
+</code></pre></div></div>
+
+<h2 id="stream-encryption">Stream Encryption</h2>
+
+<p>Our encryption is done using AES/CTR. CTR is a mode that has some very
+nice properties for us:</p>
+
+<ul>
+ <li>It is seeded so that identical data is encrypted differently.</li>
+ <li>It does not require padding the stream to the cipher length.</li>
+ <li>It allows readers to seek in to a stream.</li>
+ <li>The IV does not need to be randomly generated.</li>
+</ul>
+
+<p>To ensure that we don’t reuse IV, we set the IV as:</p>
+
+<ul>
+ <li>bytes 0 to 2 - column id</li>
+ <li>bytes 3 to 4 - stream kind</li>
+ <li>bytes 5 to 7 - stripe id</li>
+ <li>bytes 8 to 15 - cipher block counter</li>
+</ul>
+
+<p>However, it is critical for CTR that we never reuse an initialization
+vector (IV) with the same local key.</p>
+
+<p>For data in the footer, use the number of stripes in the file as the
+stripe id. This guarantees when we write an intermediate footer in to
+a file that we don’t use the same IV.</p>
+
+<p>Additionally, we never reuse a local key for new data. For example, when
+merging files, we don’t reuse local key from the input files for the new
+file tail, but always generate a new local key.</p>
+
<h1 id="compression">Compression</h1>
<p>If the ORC file writer selects a generic compression codec (zlib or
@@ -903,6 +1135,23 @@ uses three streams PRESENT, DATA, and LENGTH, which stores the length
of each value. The details of each type will be presented in the
following subsections.</p>
+<p>The layout of each stripe looks like:</p>
+<ul>
+ <li>index streams
+ <ul>
+ <li>unencrypted</li>
+ <li>encryption variant 1..N</li>
+ </ul>
+ </li>
+ <li>data streams
+ <ul>
+ <li>unencrypted</li>
+ <li>encryption variant 1..N</li>
+ </ul>
+ </li>
+ <li>stripe footer</li>
+</ul>
+
<h2 id="stripe-footer">Stripe Footer</h2>
<p>The stripe footer contains the encoding of each column and the
@@ -913,6 +1162,22 @@ directory of the streams including their location.</p>
repeated Stream streams = 1;
// the encoding of each column
repeated ColumnEncoding columns = 2;
+ optional string writerTimezone = 3;
+ // one for each column encryption variant
+ repeated StripeEncryptionVariant encryption = 4;
+}
+</code></pre></div></div>
+
+<p>If the file includes encrypted columns, those streams and column
+encodings are stored separately in a StripeEncryptionVariant per an
+encryption variant. Additionally, the StripeFooter will contain two
+additional virtual streams ENCRYPTED_INDEX and ENCRYPTED_DATA that
+allocate the space that is used by the encryption variants to store
+the encrypted index and data streams.</p>
+
+<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message StripeEncryptionVariant {
+ repeated Stream streams = 1;
+ repeated ColumnEncoding encoding = 2;
}
</code></pre></div></div>
@@ -922,26 +1187,35 @@ depends on the type and encoding of the column.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message Stream {
enum Kind {
- // boolean stream of whether the next value is non-null
- PRESENT = 0;
- // the primary data stream
- DATA = 1;
- // the length of each value for variable length data
- LENGTH = 2;
- // the dictionary blob
- DICTIONARY_DATA = 3;
- // deprecated prior to Hive 0.11
- // It was used to store the number of instances of each value in the
- // dictionary
- DICTIONARY_COUNT = 4;
- // a secondary data stream
- SECONDARY = 5;
- // the index for seeking to particular row groups
- ROW_INDEX = 6;
- // original bloom filters used before ORC-101
- BLOOM_FILTER = 7;
- // bloom filters that consistently use utf8
- BLOOM_FILTER_UTF8 = 8;
+ // boolean stream of whether the next value is non-null
+ PRESENT = 0;
+ // the primary data stream
+ DATA = 1;
+ // the length of each value for variable length data
+ LENGTH = 2;
+ // the dictionary blob
+ DICTIONARY_DATA = 3;
+ // deprecated prior to Hive 0.11
+ // It was used to store the number of instances of each value in the
+ // dictionary
+ DICTIONARY_COUNT = 4;
+ // a secondary data stream
+ SECONDARY = 5;
+ // the index for seeking to particular row groups
+ ROW_INDEX = 6;
+ // original bloom filters used before ORC-101
+ BLOOM_FILTER = 7;
+ // bloom filters that consistently use utf8
+ BLOOM_FILTER_UTF8 = 8;
+
+ // Virtual stream kinds to allocate space for encrypted index and data.
+ ENCRYPTED_INDEX = 9;
+ ENCRYPTED_DATA = 10;
+
+ // stripe statistics streams
+ STRIPE_STATISTICS = 100;
+ // A virtual stream kind that is used for setting the encryption IV.
+ FILE_STATISTICS = 101;
}
required Kind kind = 1;
// the column id