You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@orc.apache.org by do...@apache.org on 2021/04/04 19:56:06 UTC
[orc] branch branch-1.6 updated: MINOR: Fix RLEv1 encoding examples
in ORCv0/v1/v2 docs (#676)
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/orc.git
The following commit(s) were added to refs/heads/branch-1.6 by this push:
new 925030c MINOR: Fix RLEv1 encoding examples in ORCv0/v1/v2 docs (#676)
925030c is described below
commit 925030cdaa4cf8f0752417dd73bddfa9275b6681
Author: yan.zhang <di...@gmail.com>
AuthorDate: Mon Apr 5 03:55:16 2021 +0800
MINOR: Fix RLEv1 encoding examples in ORCv0/v1/v2 docs (#676)
Change the literals-encoding example to [2,3,6,7,11]. Either [2,3,4,7,11] or first 5 prime numbers [2,3,5,7,11] has run, and can not be used as a literals-encoding example.
------
The first 5 prime numbers are [2,3,5,7,11], and there is a run. The encoded data is
- 2 [0xff, 0x02]
- 3,5,7 (delta=2,length=3,base=3) [0x00, 0x02, 0x03]
- 11 [0xff, 0xb]
And if we use example [2,3,4,7,11] as the spec written, there is also a run .The encoded data is
- 2,3,4 (delta=1, length=3, base=2) [0x00, 0x01, 0x02]
- 7,11 [0xfe, 0x7, 0xb]
------
And I have verified the encoded data with the following C++ code. It used RLEEncoderv1 to encode [2,3,5,7,11] and dump data to a local file called `rle.data`.
```c++
int main() {
auto pool = getDefaultPool();
auto outputStream = writeLocalFile("rle.data");
auto buf = new BufferedOutputStream(*pool, outputStream.get(), 1024, 32);
std::unique_ptr<BufferedOutputStream> pbuf(buf);
auto encoder = new RleEncoderV1(std::move(pbuf), false);
encoder->write(2);
encoder->write(3);
encoder->write(5);
encoder->write(7);
encoder->write(11);
encoder->flush();
return 0;
}
```
And then I use `od -A n -t x1 rle.data` to see data in hex format. The result is following
```
ff 02 00 02 03 ff 0b
```
(cherry picked from commit 0336d24a980b8f10d49b1482c505f9c899a1eae7)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
site/specification/ORCv0.md | 7 +++----
site/specification/ORCv1.md | 5 ++---
site/specification/ORCv2.md | 5 ++---
3 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/site/specification/ORCv0.md b/site/specification/ORCv0.md
index 2ca58b6..3dd39cd 100644
--- a/site/specification/ORCv0.md
+++ b/site/specification/ORCv0.md
@@ -478,9 +478,9 @@ and the varint is 100 for an encoding of [0x61, 0xff, 0x64].
Literals start with an initial byte of 0x80 to 0xff, which corresponds
to the negative of number of literals in the sequence. Following the
header byte, the list of N varints is encoded. Thus, if there are
-no runs, the overhead is 1 byte for each 128 integers. The first 5
-prime numbers [2, 3, 4, 7, 11] would encoded as [0xfb, 0x02, 0x03,
-0x04, 0x07, 0xb].
+no runs, the overhead is 1 byte for each 128 integers. Numbers
+[2, 3, 6, 7, 11] would be encoded as [0xfb, 0x02, 0x03, 0x06, 0x07, 0xb].
+
# Stripes
@@ -789,4 +789,3 @@ indexes error-prone.
Because dictionaries are accessed randomly, there is not a position to
record for the dictionary and the entire dictionary must be read even
if only part of a stripe is being read.
-
diff --git a/site/specification/ORCv1.md b/site/specification/ORCv1.md
index 2bf9f96..808a80b 100644
--- a/site/specification/ORCv1.md
+++ b/site/specification/ORCv1.md
@@ -715,9 +715,8 @@ and the varint is 100 for an encoding of [0x61, 0xff, 0x64].
Literals start with an initial byte of 0x80 to 0xff, which corresponds
to the negative of number of literals in the sequence. Following the
header byte, the list of N varints is encoded. Thus, if there are
-no runs, the overhead is 1 byte for each 128 integers. The first 5
-prime numbers [2, 3, 4, 7, 11] would encoded as [0xfb, 0x02, 0x03,
-0x04, 0x07, 0xb].
+no runs, the overhead is 1 byte for each 128 integers. Numbers
+[2, 3, 6, 7, 11] would be encoded as [0xfb, 0x02, 0x03, 0x06, 0x07, 0xb].
## Integer Run Length Encoding, version 2
diff --git a/site/specification/ORCv2.md b/site/specification/ORCv2.md
index a1d46f7..6c76f68 100644
--- a/site/specification/ORCv2.md
+++ b/site/specification/ORCv2.md
@@ -734,9 +734,8 @@ and the varint is 100 for an encoding of [0x61, 0xff, 0x64].
Literals start with an initial byte of 0x80 to 0xff, which corresponds
to the negative of number of literals in the sequence. Following the
header byte, the list of N varints is encoded. Thus, if there are
-no runs, the overhead is 1 byte for each 128 integers. The first 5
-prime numbers [2, 3, 4, 7, 11] would encoded as [0xfb, 0x02, 0x03,
-0x04, 0x07, 0xb].
+no runs, the overhead is 1 byte for each 128 integers. Numbers
+[2, 3, 6, 7, 11] would be encoded as [0xfb, 0x02, 0x03, 0x06, 0x07, 0xb].
## Integer Run Length Encoding, version 2