You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@parquet.apache.org by ju...@apache.org on 2016/01/29 19:30:14 UTC
parquet-format git commit: PARQUET-450: Fix several typos in Parquet
format documentation
Repository: parquet-format
Updated Branches:
refs/heads/master c3682e3b3 -> 2a8f022a4
PARQUET-450: Fix several typos in Parquet format documentation
It also changes parquet.thrift location to conform to maven layout and add a link to it from README.md
Author: Laurent Goujon <lg...@twitter.com>
Closes #36 from laurentgo/update-format-specification and squashes the following commits:
244c119 [Laurent Goujon] Fix several typos/errors in Parquet documentation
90a2be4 [Laurent Goujon] Fix thrift source path to match maven layout
Project: http://git-wip-us.apache.org/repos/asf/parquet-format/repo
Commit: http://git-wip-us.apache.org/repos/asf/parquet-format/commit/2a8f022a
Tree: http://git-wip-us.apache.org/repos/asf/parquet-format/tree/2a8f022a
Diff: http://git-wip-us.apache.org/repos/asf/parquet-format/diff/2a8f022a
Branch: refs/heads/master
Commit: 2a8f022a49555fa7476dbb6804c9771397f8da12
Parents: c3682e3
Author: Laurent Goujon <lg...@twitter.com>
Authored: Fri Jan 29 10:30:06 2016 -0800
Committer: Julien Le Dem <ju...@dremio.com>
Committed: Fri Jan 29 10:30:06 2016 -0800
----------------------------------------------------------------------
Encodings.md | 2 +-
README.md | 27 +-
pom.xml | 2 +-
src/main/thrift/parquet.thrift | 573 ++++++++++++++++++++++++++++++++++++
src/thrift/parquet.thrift | 573 ------------------------------------
5 files changed, 589 insertions(+), 588 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/parquet-format/blob/2a8f022a/Encodings.md
----------------------------------------------------------------------
diff --git a/Encodings.md b/Encodings.md
index 42961ba..c4cdf70 100644
--- a/Encodings.md
+++ b/Encodings.md
@@ -53,7 +53,7 @@ using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too b
or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is
written first, before the data pages of the column chunk.
-Dictionary page format: the entries in the dictionary - in dictionary order - using the [plain](#PLAIN) enncoding.
+Dictionary page format: the entries in the dictionary - in dictionary order - using the [plain](#PLAIN) encoding.
Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32),
followed by the values encoded using RLE/Bit packed described above (with the given bit width).
http://git-wip-us.apache.org/repos/asf/parquet-format/blob/2a8f022a/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 904bb6a..b0423e6 100644
--- a/README.md
+++ b/README.md
@@ -46,30 +46,30 @@ The `parquet-compatibility` project contains compatibility tests that can be use
## Building
-Java resources can be build using `mvn package.` The current stable version should always be available from Maven Central.
+Java resources can be build using `mvn package`. The current stable version should always be available from Maven Central.
C++ thrift resources can be generated via make.
Thrift can be also code-genned into any other thrift-supported language.
## Glossary
- - Block (hdfs block): This means a block in hdfs and the meaning is
+ - Block (HDFS block): This means a block in HDFS and the meaning is
unchanged for describing this file format. The file format is
- designed to work well on top of hdfs.
+ designed to work well on top of HDFS.
- - File: A hdfs file that must include the metadata for the file.
+ - File: A HDFS file that must include the metadata for the file.
It does not need to actually contain the data.
- Row group: A logical horizontal partitioning of the data into rows.
There is no physical structure that is guaranteed for a row group.
A row group consists of a column chunk for each column in the dataset.
- - Column chunk: A chunk of the data for a particular column. These live
- in a particular row group and is guaranteed to be contiguous in the file.
+ - Column chunk: A chunk of the data for a particular column. They live
+ in a particular row group and are guaranteed to be contiguous in the file.
- Page: Column chunks are divided up into pages. A page is conceptually
an indivisible unit (in terms of compression and encoding). There can
- be multiple page types which is interleaved in a column chunk.
+ be multiple page types which are interleaved in a column chunk.
Hierarchically, a file consists of one or more row groups. A row group
contains exactly one column chunk per column. Column chunks contain one or
@@ -81,7 +81,7 @@ more pages.
- Encoding/Compression - Page
## File format
-This file and the thrift definition should be read together to understand the format.
+This file and the [thrift definition](src/main/thrift/parquet.thrift) should be read together to understand the format.
4-byte magic number "PAR1"
<Column 1 Chunk 1 + Column Metadata>
@@ -98,13 +98,13 @@ This file and the thrift definition should be read together to understand the fo
...
<Column N Chunk M + Column Metadata>
File Metadata
- 4-byte length in bytes of file metadata
+ 4-byte length in bytes of file metadata (little endian)
4-byte magic number "PAR1"
In the above example, there are N columns in this table, split into M row
groups. The file metadata contains the locations of all the column metadata
start locations. More details on what is contained in the metadata can be found
-in the thrift files.
+in the thrift definition.
Metadata is written after the data to allow for single pass writing.
@@ -139,7 +139,7 @@ by specifying how the primitive types should be interpreted. This keeps the set
of primitive types to a minimum and reuses parquet's efficient encodings. For
example, strings are stored as byte arrays (binary) with a UTF8 annotation.
These annotations define how to further decode and interpret the data.
-Annotations are stored as a `ConvertedType` in the file metadata and are
+Annotations are stored as `ConvertedType` fields in the file metadata and are
documented in
[LogicalTypes.md][logical-types].
@@ -165,9 +165,10 @@ nothing else.
## Data Pages
For data pages, the 3 pieces of information are encoded back to back, after the page
header. We have the
- - definition levels data,
- repetition levels data,
+ - definition levels data,
- encoded values.
+
The size of specified in the header is for all 3 pieces combined.
The data for the data page is always required. The definition and repetition levels
@@ -183,7 +184,7 @@ The supported encodings are described in [Encodings.md](https://github.com/Parqu
## Column chunks
Column chunks are composed of pages written back to back. The pages share a common
-header and readers can skip over page they are not interested in. The data for the
+header and readers can skip over pages they are not interested in. The data for the
page follows the header and can be compressed and/or encoded. The compression and
encoding is specified in the page metadata.
http://git-wip-us.apache.org/repos/asf/parquet-format/blob/2a8f022a/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 78578b9..d5b3b60 100644
--- a/pom.xml
+++ b/pom.xml
@@ -96,7 +96,7 @@
<artifactId>maven-thrift-plugin</artifactId>
<version>0.1.11</version>
<configuration>
- <thriftSourceRoot>src/thrift</thriftSourceRoot>
+ <thriftSourceRoot>src/main/thrift</thriftSourceRoot>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/parquet-format/blob/2a8f022a/src/main/thrift/parquet.thrift
----------------------------------------------------------------------
diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift
new file mode 100644
index 0000000..d1b305e
--- /dev/null
+++ b/src/main/thrift/parquet.thrift
@@ -0,0 +1,573 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * File format description for the parquet file format
+ */
+namespace cpp parquet
+namespace java org.apache.parquet.format
+
+/**
+ * Types supported by Parquet. These types are intended to be used in combination
+ * with the encodings to control the on disk storage format.
+ * For example INT16 is not included as a type since a good encoding of INT32
+ * would handle this.
+ */
+enum Type {
+ BOOLEAN = 0;
+ INT32 = 1;
+ INT64 = 2;
+ INT96 = 3;
+ FLOAT = 4;
+ DOUBLE = 5;
+ BYTE_ARRAY = 6;
+ FIXED_LEN_BYTE_ARRAY = 7;
+}
+
+/**
+ * Common types used by frameworks(e.g. hive, pig) using parquet. This helps map
+ * between types in those frameworks to the base types in parquet. This is only
+ * metadata and not needed to read or write the data.
+ */
+enum ConvertedType {
+ /** a BYTE_ARRAY actually contains UTF8 encoded chars */
+ UTF8 = 0;
+
+ /** a map is converted as an optional field containing a repeated key/value pair */
+ MAP = 1;
+
+ /** a key/value pair is converted into a group of two fields */
+ MAP_KEY_VALUE = 2;
+
+ /** a list is converted into an optional field containing a repeated field for its
+ * values */
+ LIST = 3;
+
+ /** an enum is converted into a binary field */
+ ENUM = 4;
+
+ /**
+ * A decimal value.
+ *
+ * This may be used to annotate binary or fixed primitive types. The
+ * underlying byte array stores the unscaled value encoded as two's
+ * complement using big-endian byte order (the most significant byte is the
+ * zeroth element). The value of the decimal is the value * 10^{-scale}.
+ *
+ * This must be accompanied by a (maximum) precision and a scale in the
+ * SchemaElement. The precision specifies the number of digits in the decimal
+ * and the scale stores the location of the decimal point. For example 1.23
+ * would have precision 3 (3 total digits) and scale 2 (the decimal point is
+ * 2 digits over).
+ */
+ DECIMAL = 5;
+
+ /**
+ * A Date
+ *
+ * Stored as days since Unix epoch, encoded as the INT32 physical type.
+ *
+ */
+ DATE = 6;
+
+ /**
+ * A time
+ *
+ * The total number of milliseconds since midnight. The value is stored
+ * as an INT32 physical type.
+ */
+ TIME_MILLIS = 7;
+
+ /**
+ * A time.
+ *
+ * The total number of microseconds since midnight. The value is stored as
+ * an INT64 physical type.
+ */
+ TIME_MICROS = 8;
+
+ /**
+ * A date/time combination
+ *
+ * Date and time recorded as milliseconds since the Unix epoch. Recorded as
+ * a physical type of INT64.
+ */
+ TIMESTAMP_MILLIS = 9;
+
+ /**
+ * A date/time combination
+ *
+ * Date and time recorded as microseconds since the Unix epoch. The value is
+ * stored as an INT64 physical type.
+ */
+ TIMESTAMP_MICROS = 10;
+
+
+ /**
+ * An unsigned integer value.
+ *
+ * The number describes the maximum number of meainful data bits in
+ * the stored value. 8, 16 and 32 bit values are stored using the
+ * INT32 physical type. 64 bit values are stored using the INT64
+ * physical type.
+ *
+ */
+ UINT_8 = 11;
+ UINT_16 = 12;
+ UINT_32 = 13;
+ UINT_64 = 14;
+
+ /**
+ * A signed integer value.
+ *
+ * The number describes the maximum number of meainful data bits in
+ * the stored value. 8, 16 and 32 bit values are stored using the
+ * INT32 physical type. 64 bit values are stored using the INT64
+ * physical type.
+ *
+ */
+ INT_8 = 15;
+ INT_16 = 16;
+ INT_32 = 17;
+ INT_64 = 18;
+
+ /**
+ * An embedded JSON document
+ *
+ * A JSON document embedded within a single UTF8 column.
+ */
+ JSON = 19;
+
+ /**
+ * An embedded BSON document
+ *
+ * A BSON document embedded within a single BINARY column.
+ */
+ BSON = 20;
+
+ /**
+ * An interval of time
+ *
+ * This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 12
+ * This data is composed of three separate little endian unsigned
+ * integers. Each stores a component of a duration of time. The first
+ * integer identifies the number of months associated with the duration,
+ * the second identifies the number of days associated with the duration
+ * and the third identifies the number of milliseconds associated with
+ * the provided duration. This duration of time is independent of any
+ * particular timezone or date.
+ */
+ INTERVAL = 21;
+
+}
+
+/**
+ * Representation of Schemas
+ */
+enum FieldRepetitionType {
+ /** This field is required (can not be null) and each record has exactly 1 value. */
+ REQUIRED = 0;
+
+ /** The field is optional (can be null) and each record has 0 or 1 values. */
+ OPTIONAL = 1;
+
+ /** The field is repeated and can contain 0 or more values */
+ REPEATED = 2;
+}
+
+/**
+ * Statistics per row group and per page
+ * All fields are optional.
+ */
+struct Statistics {
+ /** min and max value of the column, encoded in PLAIN encoding */
+ 1: optional binary max;
+ 2: optional binary min;
+ /** count of null value in the column */
+ 3: optional i64 null_count;
+ /** count of distinct values occurring */
+ 4: optional i64 distinct_count;
+}
+
+/**
+ * Represents a element inside a schema definition.
+ * - if it is a group (inner node) then type is undefined and num_children is defined
+ * - if it is a primitive type (leaf) then type is defined and num_children is undefined
+ * the nodes are listed in depth first traversal order.
+ */
+struct SchemaElement {
+ /** Data type for this field. Not set if the current element is a non-leaf node */
+ 1: optional Type type;
+
+ /** If type is FIXED_LEN_BYTE_ARRAY, this is the byte length of the vales.
+ * Otherwise, if specified, this is the maximum bit length to store any of the values.
+ * (e.g. a low cardinality INT col could have this set to 3). Note that this is
+ * in the schema, and therefore fixed for the entire file.
+ */
+ 2: optional i32 type_length;
+
+ /** repetition of the field. The root of the schema does not have a repetition_type.
+ * All other nodes must have one */
+ 3: optional FieldRepetitionType repetition_type;
+
+ /** Name of the field in the schema */
+ 4: required string name;
+
+ /** Nested fields. Since thrift does not support nested fields,
+ * the nesting is flattened to a single list by a depth-first traversal.
+ * The children count is used to construct the nested relationship.
+ * This field is not set when the element is a primitive type
+ */
+ 5: optional i32 num_children;
+
+ /** When the schema is the result of a conversion from another model
+ * Used to record the original type to help with cross conversion.
+ */
+ 6: optional ConvertedType converted_type;
+
+ /** Used when this column contains decimal data.
+ * See the DECIMAL converted type for more details.
+ */
+ 7: optional i32 scale
+ 8: optional i32 precision
+
+ /** When the original schema supports field ids, this will save the
+ * original field id in the parquet schema
+ */
+ 9: optional i32 field_id;
+
+}
+
+/**
+ * Encodings supported by Parquet. Not all encodings are valid for all types. These
+ * enums are also used to specify the encoding of definition and repetition levels.
+ * See the accompanying doc for the details of the more complicated encodings.
+ */
+enum Encoding {
+ /** Default encoding.
+ * BOOLEAN - 1 bit per value. 0 is false; 1 is true.
+ * INT32 - 4 bytes per value. Stored as little-endian.
+ * INT64 - 8 bytes per value. Stored as little-endian.
+ * FLOAT - 4 bytes per value. IEEE. Stored as little-endian.
+ * DOUBLE - 8 bytes per value. IEEE. Stored as little-endian.
+ * BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes.
+ * FIXED_LEN_BYTE_ARRAY - Just the bytes.
+ */
+ PLAIN = 0;
+
+ /** Group VarInt encoding for INT32/INT64.
+ * This encoding is deprecated. It was never used
+ */
+ // GROUP_VAR_INT = 1;
+
+ /**
+ * Deprecated: Dictionary encoding. The values in the dictionary are encoded in the
+ * plain type.
+ * in a data page use RLE_DICTIONARY instead.
+ * in a Dictionary page use PLAIN instead
+ */
+ PLAIN_DICTIONARY = 2;
+
+ /** Group packed run length encoding. Usable for definition/reptition levels
+ * encoding and Booleans (on one bit: 0 is false; 1 is true.)
+ */
+ RLE = 3;
+
+ /** Bit packed encoding. This can only be used if the data has a known max
+ * width. Usable for definition/repetition levels encoding.
+ */
+ BIT_PACKED = 4;
+
+ /** Delta encoding for integers. This can be used for int columns and works best
+ * on sorted data
+ */
+ DELTA_BINARY_PACKED = 5;
+
+ /** Encoding for byte arrays to separate the length values and the data. The lengths
+ * are encoded using DELTA_BINARY_PACKED
+ */
+ DELTA_LENGTH_BYTE_ARRAY = 6;
+
+ /** Incremental-encoded byte array. Prefix lengths are encoded using DELTA_BINARY_PACKED.
+ * Suffixes are stored as delta length byte arrays.
+ */
+ DELTA_BYTE_ARRAY = 7;
+
+ /** Dictionary encoding: the ids are encoded using the RLE encoding
+ */
+ RLE_DICTIONARY = 8;
+}
+
+/**
+ * Supported compression algorithms.
+ */
+enum CompressionCodec {
+ UNCOMPRESSED = 0;
+ SNAPPY = 1;
+ GZIP = 2;
+ LZO = 3;
+}
+
+enum PageType {
+ DATA_PAGE = 0;
+ INDEX_PAGE = 1;
+ DICTIONARY_PAGE = 2;
+ DATA_PAGE_V2 = 3;
+}
+
+/** Data page header */
+struct DataPageHeader {
+ /** Number of values, including NULLs, in this data page. **/
+ 1: required i32 num_values
+
+ /** Encoding used for this data page **/
+ 2: required Encoding encoding
+
+ /** Encoding used for definition levels **/
+ 3: required Encoding definition_level_encoding;
+
+ /** Encoding used for repetition levels **/
+ 4: required Encoding repetition_level_encoding;
+
+ /** Optional statistics for the data in this page**/
+ 5: optional Statistics statistics;
+}
+
+struct IndexPageHeader {
+ /** TODO: **/
+}
+
+struct DictionaryPageHeader {
+ /** Number of values in the dictionary **/
+ 1: required i32 num_values;
+
+ /** Encoding using this dictionary page **/
+ 2: required Encoding encoding
+
+ /** If true, the entries in the dictionary are sorted in ascending order **/
+ 3: optional bool is_sorted;
+}
+
+/**
+ * New page format alowing reading levels without decompressing the data
+ * Repetition and definition levels are uncompressed
+ * The remaining section containing the data is compressed if is_compressed is true
+ **/
+struct DataPageHeaderV2 {
+ /** Number of values, including NULLs, in this data page. **/
+ 1: required i32 num_values
+ /** Number of NULL values, in this data page.
+ Number of non-null = num_values - num_nulls which is also the number of values in the data section **/
+ 2: required i32 num_nulls
+ /** Number of rows in this data page. which means pages change on record boundaries (r = 0) **/
+ 3: required i32 num_rows
+ /** Encoding used for data in this page **/
+ 4: required Encoding encoding
+
+ // repetition levels and definition levels are always using RLE (without size in it)
+
+ /** length of the repetition levels */
+ 5: required i32 definition_levels_byte_length;
+ /** length of the definition levels */
+ 6: required i32 repetition_levels_byte_length;
+
+ /** whether the values are compressed.
+ Which means the section of the page between
+ definition_levels_byte_length + repetition_levels_byte_length + 1 and compressed_page_size (included)
+ is compressed with the compression_codec.
+ If missing it is considered compressed */
+ 7: optional bool is_compressed = 1;
+
+ /** optional statistics for this column chunk */
+ 8: optional Statistics statistics;
+}
+
+struct PageHeader {
+ /** the type of the page: indicates which of the *_header fields is set **/
+ 1: required PageType type
+
+ /** Uncompressed page size in bytes (not including this header) **/
+ 2: required i32 uncompressed_page_size
+
+ /** Compressed page size in bytes (not including this header) **/
+ 3: required i32 compressed_page_size
+
+ /** 32bit crc for the data below. This allows for disabling checksumming in HDFS
+ * if only a few pages needs to be read
+ **/
+ 4: optional i32 crc
+
+ // Headers for page specific data. One only will be set.
+ 5: optional DataPageHeader data_page_header;
+ 6: optional IndexPageHeader index_page_header;
+ 7: optional DictionaryPageHeader dictionary_page_header;
+ 8: optional DataPageHeaderV2 data_page_header_v2;
+}
+
+/**
+ * Wrapper struct to store key values
+ */
+ struct KeyValue {
+ 1: required string key
+ 2: optional string value
+}
+
+/**
+ * Wrapper struct to specify sort order
+ */
+struct SortingColumn {
+ /** The column index (in this row group) **/
+ 1: required i32 column_idx
+
+ /** If true, indicates this column is sorted in descending order. **/
+ 2: required bool descending
+
+ /** If true, nulls will come before non-null values, otherwise,
+ * nulls go at the end. */
+ 3: required bool nulls_first
+}
+
+/**
+ * statistics of a given page type and encoding
+ */
+struct PageEncodingStats {
+
+ /** the page type (data/dic/...) **/
+ 1: required PageType page_type;
+
+ /** encoding of the page **/
+ 2: required Encoding encoding;
+
+ /** number of pages of this type with this encoding **/
+ 3: required i32 count;
+
+}
+
+/**
+ * Description for column metadata
+ */
+struct ColumnMetaData {
+ /** Type of this column **/
+ 1: required Type type
+
+ /** Set of all encodings used for this column. The purpose is to validate
+ * whether we can decode those pages. **/
+ 2: required list<Encoding> encodings
+
+ /** Path in schema **/
+ 3: required list<string> path_in_schema
+
+ /** Compression codec **/
+ 4: required CompressionCodec codec
+
+ /** Number of values in this column **/
+ 5: required i64 num_values
+
+ /** total byte size of all uncompressed pages in this column chunk (including the headers) **/
+ 6: required i64 total_uncompressed_size
+
+ /** total byte size of all compressed pages in this column chunk (including the headers) **/
+ 7: required i64 total_compressed_size
+
+ /** Optional key/value metadata **/
+ 8: optional list<KeyValue> key_value_metadata
+
+ /** Byte offset from beginning of file to first data page **/
+ 9: required i64 data_page_offset
+
+ /** Byte offset from beginning of file to root index page **/
+ 10: optional i64 index_page_offset
+
+ /** Byte offset from the beginning of file to first (only) dictionary page **/
+ 11: optional i64 dictionary_page_offset
+
+ /** optional statistics for this column chunk */
+ 12: optional Statistics statistics;
+
+ /** Set of all encodings used for pages in this column chunk.
+ * This information can be used to determine if all data pages are
+ * dictionary encoded for example **/
+ 13: optional list<PageEncodingStats> encoding_stats;
+}
+
+struct ColumnChunk {
+ /** File where column data is stored. If not set, assumed to be same file as
+ * metadata. This path is relative to the current file.
+ **/
+ 1: optional string file_path
+
+ /** Byte offset in file_path to the ColumnMetaData **/
+ 2: required i64 file_offset
+
+ /** Column metadata for this chunk. This is the same content as what is at
+ * file_path/file_offset. Having it here has it replicated in the file
+ * metadata.
+ **/
+ 3: optional ColumnMetaData meta_data
+}
+
+struct RowGroup {
+ /** Metadata for each column chunk in this row group.
+ * This list must have the same order as the SchemaElement list in FileMetaData.
+ **/
+ 1: required list<ColumnChunk> columns
+
+ /** Total byte size of all the uncompressed column data in this row group **/
+ 2: required i64 total_byte_size
+
+ /** Number of rows in this row group **/
+ 3: required i64 num_rows
+
+ /** If set, specifies a sort ordering of the rows in this RowGroup.
+ * The sorting columns can be a subset of all the columns.
+ */
+ 4: optional list<SortingColumn> sorting_columns
+}
+
+/**
+ * Description for file metadata
+ */
+struct FileMetaData {
+ /** Version of this file **/
+ 1: required i32 version
+
+ /** Parquet schema for this file. This schema contains metadata for all the columns.
+ * The schema is represented as a tree with a single root. The nodes of the tree
+ * are flattened to a list by doing a depth-first traversal.
+ * The column metadata contains the path in the schema for that column which can be
+ * used to map columns to nodes in the schema.
+ * The first element is the root **/
+ 2: required list<SchemaElement> schema;
+
+ /** Number of rows in this file **/
+ 3: required i64 num_rows
+
+ /** Row groups in this file **/
+ 4: required list<RowGroup> row_groups
+
+ /** Optional key/value metadata **/
+ 5: optional list<KeyValue> key_value_metadata
+
+ /** String for application that wrote this file. This should be in the format
+ * <Application> version <App Version> (build <App Build Hash>).
+ * e.g. impala version 1.0 (build 6cf94d29b2b7115df4de2c06e2ab4326d721eb55)
+ **/
+ 6: optional string created_by
+}
+
http://git-wip-us.apache.org/repos/asf/parquet-format/blob/2a8f022a/src/thrift/parquet.thrift
----------------------------------------------------------------------
diff --git a/src/thrift/parquet.thrift b/src/thrift/parquet.thrift
deleted file mode 100644
index d1b305e..0000000
--- a/src/thrift/parquet.thrift
+++ /dev/null
@@ -1,573 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * File format description for the parquet file format
- */
-namespace cpp parquet
-namespace java org.apache.parquet.format
-
-/**
- * Types supported by Parquet. These types are intended to be used in combination
- * with the encodings to control the on disk storage format.
- * For example INT16 is not included as a type since a good encoding of INT32
- * would handle this.
- */
-enum Type {
- BOOLEAN = 0;
- INT32 = 1;
- INT64 = 2;
- INT96 = 3;
- FLOAT = 4;
- DOUBLE = 5;
- BYTE_ARRAY = 6;
- FIXED_LEN_BYTE_ARRAY = 7;
-}
-
-/**
- * Common types used by frameworks(e.g. hive, pig) using parquet. This helps map
- * between types in those frameworks to the base types in parquet. This is only
- * metadata and not needed to read or write the data.
- */
-enum ConvertedType {
- /** a BYTE_ARRAY actually contains UTF8 encoded chars */
- UTF8 = 0;
-
- /** a map is converted as an optional field containing a repeated key/value pair */
- MAP = 1;
-
- /** a key/value pair is converted into a group of two fields */
- MAP_KEY_VALUE = 2;
-
- /** a list is converted into an optional field containing a repeated field for its
- * values */
- LIST = 3;
-
- /** an enum is converted into a binary field */
- ENUM = 4;
-
- /**
- * A decimal value.
- *
- * This may be used to annotate binary or fixed primitive types. The
- * underlying byte array stores the unscaled value encoded as two's
- * complement using big-endian byte order (the most significant byte is the
- * zeroth element). The value of the decimal is the value * 10^{-scale}.
- *
- * This must be accompanied by a (maximum) precision and a scale in the
- * SchemaElement. The precision specifies the number of digits in the decimal
- * and the scale stores the location of the decimal point. For example 1.23
- * would have precision 3 (3 total digits) and scale 2 (the decimal point is
- * 2 digits over).
- */
- DECIMAL = 5;
-
- /**
- * A Date
- *
- * Stored as days since Unix epoch, encoded as the INT32 physical type.
- *
- */
- DATE = 6;
-
- /**
- * A time
- *
- * The total number of milliseconds since midnight. The value is stored
- * as an INT32 physical type.
- */
- TIME_MILLIS = 7;
-
- /**
- * A time.
- *
- * The total number of microseconds since midnight. The value is stored as
- * an INT64 physical type.
- */
- TIME_MICROS = 8;
-
- /**
- * A date/time combination
- *
- * Date and time recorded as milliseconds since the Unix epoch. Recorded as
- * a physical type of INT64.
- */
- TIMESTAMP_MILLIS = 9;
-
- /**
- * A date/time combination
- *
- * Date and time recorded as microseconds since the Unix epoch. The value is
- * stored as an INT64 physical type.
- */
- TIMESTAMP_MICROS = 10;
-
-
- /**
- * An unsigned integer value.
- *
- * The number describes the maximum number of meainful data bits in
- * the stored value. 8, 16 and 32 bit values are stored using the
- * INT32 physical type. 64 bit values are stored using the INT64
- * physical type.
- *
- */
- UINT_8 = 11;
- UINT_16 = 12;
- UINT_32 = 13;
- UINT_64 = 14;
-
- /**
- * A signed integer value.
- *
- * The number describes the maximum number of meainful data bits in
- * the stored value. 8, 16 and 32 bit values are stored using the
- * INT32 physical type. 64 bit values are stored using the INT64
- * physical type.
- *
- */
- INT_8 = 15;
- INT_16 = 16;
- INT_32 = 17;
- INT_64 = 18;
-
- /**
- * An embedded JSON document
- *
- * A JSON document embedded within a single UTF8 column.
- */
- JSON = 19;
-
- /**
- * An embedded BSON document
- *
- * A BSON document embedded within a single BINARY column.
- */
- BSON = 20;
-
- /**
- * An interval of time
- *
- * This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 12
- * This data is composed of three separate little endian unsigned
- * integers. Each stores a component of a duration of time. The first
- * integer identifies the number of months associated with the duration,
- * the second identifies the number of days associated with the duration
- * and the third identifies the number of milliseconds associated with
- * the provided duration. This duration of time is independent of any
- * particular timezone or date.
- */
- INTERVAL = 21;
-
-}
-
-/**
- * Representation of Schemas
- */
-enum FieldRepetitionType {
- /** This field is required (can not be null) and each record has exactly 1 value. */
- REQUIRED = 0;
-
- /** The field is optional (can be null) and each record has 0 or 1 values. */
- OPTIONAL = 1;
-
- /** The field is repeated and can contain 0 or more values */
- REPEATED = 2;
-}
-
-/**
- * Statistics per row group and per page
- * All fields are optional.
- */
-struct Statistics {
- /** min and max value of the column, encoded in PLAIN encoding */
- 1: optional binary max;
- 2: optional binary min;
- /** count of null value in the column */
- 3: optional i64 null_count;
- /** count of distinct values occurring */
- 4: optional i64 distinct_count;
-}
-
-/**
- * Represents a element inside a schema definition.
- * - if it is a group (inner node) then type is undefined and num_children is defined
- * - if it is a primitive type (leaf) then type is defined and num_children is undefined
- * the nodes are listed in depth first traversal order.
- */
-struct SchemaElement {
- /** Data type for this field. Not set if the current element is a non-leaf node */
- 1: optional Type type;
-
- /** If type is FIXED_LEN_BYTE_ARRAY, this is the byte length of the vales.
- * Otherwise, if specified, this is the maximum bit length to store any of the values.
- * (e.g. a low cardinality INT col could have this set to 3). Note that this is
- * in the schema, and therefore fixed for the entire file.
- */
- 2: optional i32 type_length;
-
- /** repetition of the field. The root of the schema does not have a repetition_type.
- * All other nodes must have one */
- 3: optional FieldRepetitionType repetition_type;
-
- /** Name of the field in the schema */
- 4: required string name;
-
- /** Nested fields. Since thrift does not support nested fields,
- * the nesting is flattened to a single list by a depth-first traversal.
- * The children count is used to construct the nested relationship.
- * This field is not set when the element is a primitive type
- */
- 5: optional i32 num_children;
-
- /** When the schema is the result of a conversion from another model
- * Used to record the original type to help with cross conversion.
- */
- 6: optional ConvertedType converted_type;
-
- /** Used when this column contains decimal data.
- * See the DECIMAL converted type for more details.
- */
- 7: optional i32 scale
- 8: optional i32 precision
-
- /** When the original schema supports field ids, this will save the
- * original field id in the parquet schema
- */
- 9: optional i32 field_id;
-
-}
-
-/**
- * Encodings supported by Parquet. Not all encodings are valid for all types. These
- * enums are also used to specify the encoding of definition and repetition levels.
- * See the accompanying doc for the details of the more complicated encodings.
- */
-enum Encoding {
- /** Default encoding.
- * BOOLEAN - 1 bit per value. 0 is false; 1 is true.
- * INT32 - 4 bytes per value. Stored as little-endian.
- * INT64 - 8 bytes per value. Stored as little-endian.
- * FLOAT - 4 bytes per value. IEEE. Stored as little-endian.
- * DOUBLE - 8 bytes per value. IEEE. Stored as little-endian.
- * BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes.
- * FIXED_LEN_BYTE_ARRAY - Just the bytes.
- */
- PLAIN = 0;
-
- /** Group VarInt encoding for INT32/INT64.
- * This encoding is deprecated. It was never used
- */
- // GROUP_VAR_INT = 1;
-
- /**
- * Deprecated: Dictionary encoding. The values in the dictionary are encoded in the
- * plain type.
- * in a data page use RLE_DICTIONARY instead.
- * in a Dictionary page use PLAIN instead
- */
- PLAIN_DICTIONARY = 2;
-
- /** Group packed run length encoding. Usable for definition/reptition levels
- * encoding and Booleans (on one bit: 0 is false; 1 is true.)
- */
- RLE = 3;
-
- /** Bit packed encoding. This can only be used if the data has a known max
- * width. Usable for definition/repetition levels encoding.
- */
- BIT_PACKED = 4;
-
- /** Delta encoding for integers. This can be used for int columns and works best
- * on sorted data
- */
- DELTA_BINARY_PACKED = 5;
-
- /** Encoding for byte arrays to separate the length values and the data. The lengths
- * are encoded using DELTA_BINARY_PACKED
- */
- DELTA_LENGTH_BYTE_ARRAY = 6;
-
- /** Incremental-encoded byte array. Prefix lengths are encoded using DELTA_BINARY_PACKED.
- * Suffixes are stored as delta length byte arrays.
- */
- DELTA_BYTE_ARRAY = 7;
-
- /** Dictionary encoding: the ids are encoded using the RLE encoding
- */
- RLE_DICTIONARY = 8;
-}
-
-/**
- * Supported compression algorithms.
- */
-enum CompressionCodec {
- UNCOMPRESSED = 0;
- SNAPPY = 1;
- GZIP = 2;
- LZO = 3;
-}
-
-enum PageType {
- DATA_PAGE = 0;
- INDEX_PAGE = 1;
- DICTIONARY_PAGE = 2;
- DATA_PAGE_V2 = 3;
-}
-
-/** Data page header */
-struct DataPageHeader {
- /** Number of values, including NULLs, in this data page. **/
- 1: required i32 num_values
-
- /** Encoding used for this data page **/
- 2: required Encoding encoding
-
- /** Encoding used for definition levels **/
- 3: required Encoding definition_level_encoding;
-
- /** Encoding used for repetition levels **/
- 4: required Encoding repetition_level_encoding;
-
- /** Optional statistics for the data in this page**/
- 5: optional Statistics statistics;
-}
-
-struct IndexPageHeader {
- /** TODO: **/
-}
-
-struct DictionaryPageHeader {
- /** Number of values in the dictionary **/
- 1: required i32 num_values;
-
- /** Encoding using this dictionary page **/
- 2: required Encoding encoding
-
- /** If true, the entries in the dictionary are sorted in ascending order **/
- 3: optional bool is_sorted;
-}
-
-/**
- * New page format alowing reading levels without decompressing the data
- * Repetition and definition levels are uncompressed
- * The remaining section containing the data is compressed if is_compressed is true
- **/
-struct DataPageHeaderV2 {
- /** Number of values, including NULLs, in this data page. **/
- 1: required i32 num_values
- /** Number of NULL values, in this data page.
- Number of non-null = num_values - num_nulls which is also the number of values in the data section **/
- 2: required i32 num_nulls
- /** Number of rows in this data page. which means pages change on record boundaries (r = 0) **/
- 3: required i32 num_rows
- /** Encoding used for data in this page **/
- 4: required Encoding encoding
-
- // repetition levels and definition levels are always using RLE (without size in it)
-
- /** length of the repetition levels */
- 5: required i32 definition_levels_byte_length;
- /** length of the definition levels */
- 6: required i32 repetition_levels_byte_length;
-
- /** whether the values are compressed.
- Which means the section of the page between
- definition_levels_byte_length + repetition_levels_byte_length + 1 and compressed_page_size (included)
- is compressed with the compression_codec.
- If missing it is considered compressed */
- 7: optional bool is_compressed = 1;
-
- /** optional statistics for this column chunk */
- 8: optional Statistics statistics;
-}
-
-struct PageHeader {
- /** the type of the page: indicates which of the *_header fields is set **/
- 1: required PageType type
-
- /** Uncompressed page size in bytes (not including this header) **/
- 2: required i32 uncompressed_page_size
-
- /** Compressed page size in bytes (not including this header) **/
- 3: required i32 compressed_page_size
-
- /** 32bit crc for the data below. This allows for disabling checksumming in HDFS
- * if only a few pages needs to be read
- **/
- 4: optional i32 crc
-
- // Headers for page specific data. One only will be set.
- 5: optional DataPageHeader data_page_header;
- 6: optional IndexPageHeader index_page_header;
- 7: optional DictionaryPageHeader dictionary_page_header;
- 8: optional DataPageHeaderV2 data_page_header_v2;
-}
-
-/**
- * Wrapper struct to store key values
- */
- struct KeyValue {
- 1: required string key
- 2: optional string value
-}
-
-/**
- * Wrapper struct to specify sort order
- */
-struct SortingColumn {
- /** The column index (in this row group) **/
- 1: required i32 column_idx
-
- /** If true, indicates this column is sorted in descending order. **/
- 2: required bool descending
-
- /** If true, nulls will come before non-null values, otherwise,
- * nulls go at the end. */
- 3: required bool nulls_first
-}
-
-/**
- * statistics of a given page type and encoding
- */
-struct PageEncodingStats {
-
- /** the page type (data/dic/...) **/
- 1: required PageType page_type;
-
- /** encoding of the page **/
- 2: required Encoding encoding;
-
- /** number of pages of this type with this encoding **/
- 3: required i32 count;
-
-}
-
-/**
- * Description for column metadata
- */
-struct ColumnMetaData {
- /** Type of this column **/
- 1: required Type type
-
- /** Set of all encodings used for this column. The purpose is to validate
- * whether we can decode those pages. **/
- 2: required list<Encoding> encodings
-
- /** Path in schema **/
- 3: required list<string> path_in_schema
-
- /** Compression codec **/
- 4: required CompressionCodec codec
-
- /** Number of values in this column **/
- 5: required i64 num_values
-
- /** total byte size of all uncompressed pages in this column chunk (including the headers) **/
- 6: required i64 total_uncompressed_size
-
- /** total byte size of all compressed pages in this column chunk (including the headers) **/
- 7: required i64 total_compressed_size
-
- /** Optional key/value metadata **/
- 8: optional list<KeyValue> key_value_metadata
-
- /** Byte offset from beginning of file to first data page **/
- 9: required i64 data_page_offset
-
- /** Byte offset from beginning of file to root index page **/
- 10: optional i64 index_page_offset
-
- /** Byte offset from the beginning of file to first (only) dictionary page **/
- 11: optional i64 dictionary_page_offset
-
- /** optional statistics for this column chunk */
- 12: optional Statistics statistics;
-
- /** Set of all encodings used for pages in this column chunk.
- * This information can be used to determine if all data pages are
- * dictionary encoded for example **/
- 13: optional list<PageEncodingStats> encoding_stats;
-}
-
-struct ColumnChunk {
- /** File where column data is stored. If not set, assumed to be same file as
- * metadata. This path is relative to the current file.
- **/
- 1: optional string file_path
-
- /** Byte offset in file_path to the ColumnMetaData **/
- 2: required i64 file_offset
-
- /** Column metadata for this chunk. This is the same content as what is at
- * file_path/file_offset. Having it here has it replicated in the file
- * metadata.
- **/
- 3: optional ColumnMetaData meta_data
-}
-
-struct RowGroup {
- /** Metadata for each column chunk in this row group.
- * This list must have the same order as the SchemaElement list in FileMetaData.
- **/
- 1: required list<ColumnChunk> columns
-
- /** Total byte size of all the uncompressed column data in this row group **/
- 2: required i64 total_byte_size
-
- /** Number of rows in this row group **/
- 3: required i64 num_rows
-
- /** If set, specifies a sort ordering of the rows in this RowGroup.
- * The sorting columns can be a subset of all the columns.
- */
- 4: optional list<SortingColumn> sorting_columns
-}
-
-/**
- * Description for file metadata
- */
-struct FileMetaData {
- /** Version of this file **/
- 1: required i32 version
-
- /** Parquet schema for this file. This schema contains metadata for all the columns.
- * The schema is represented as a tree with a single root. The nodes of the tree
- * are flattened to a list by doing a depth-first traversal.
- * The column metadata contains the path in the schema for that column which can be
- * used to map columns to nodes in the schema.
- * The first element is the root **/
- 2: required list<SchemaElement> schema;
-
- /** Number of rows in this file **/
- 3: required i64 num_rows
-
- /** Row groups in this file **/
- 4: required list<RowGroup> row_groups
-
- /** Optional key/value metadata **/
- 5: optional list<KeyValue> key_value_metadata
-
- /** String for application that wrote this file. This should be in the format
- * <Application> version <App Version> (build <App Build Hash>).
- * e.g. impala version 1.0 (build 6cf94d29b2b7115df4de2c06e2ab4326d721eb55)
- **/
- 6: optional string created_by
-}
-