You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by da...@apache.org on 2018/01/29 18:26:57 UTC
kudu git commit: back out cblock nomenclature
Repository: kudu
Updated Branches:
refs/heads/master cd7166cb7 -> 9941b56bf
back out cblock nomenclature
Change-Id: I09978aa7c13a2db33df2a225faec8df9bbec001e
Reviewed-on: http://gerrit.cloudera.org:8080/9144
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/9941b56b
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/9941b56b
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/9941b56b
Branch: refs/heads/master
Commit: 9941b56bfc0610302a10cc887efe7ada5e6e14c6
Parents: cd7166c
Author: Dan Burkert <da...@apache.org>
Authored: Thu Jan 25 17:58:07 2018 -0800
Committer: Dan Burkert <da...@cloudera.com>
Committed: Mon Jan 29 18:23:01 2018 +0000
----------------------------------------------------------------------
docs/design-docs/cfile.md | 99 +++++++++++++-------------
src/kudu/cfile/cfile_reader.cc | 4 +-
src/kudu/common/iterator_stats.cc | 12 ++--
src/kudu/common/iterator_stats.h | 2 +-
src/kudu/common/wire_protocol.cc | 8 +--
src/kudu/tablet/cfile_set-test.cc | 10 +--
src/kudu/tablet/deltafile.cc | 2 +-
src/kudu/tablet/tablet-pushdown-test.cc | 8 +--
src/kudu/tserver/tserver_path_handlers.cc | 4 +-
www/scans.mustache | 4 +-
10 files changed, 76 insertions(+), 77 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/docs/design-docs/cfile.md
----------------------------------------------------------------------
diff --git a/docs/design-docs/cfile.md b/docs/design-docs/cfile.md
index 330385f..59cad6b 100644
--- a/docs/design-docs/cfile.md
+++ b/docs/design-docs/cfile.md
@@ -23,22 +23,22 @@ ad-hoc index will be stored in a separate CFile.
Despite the name, a CFile does not necessarily correspond one-to-one with a file
in the filesystem. The mapping of CFiles to the filesystem is determined by the
[BlockManager](../../src/kudu/fs/block_manager.h) abstraction. A CFile is
-written to a single BlockManager Block (not to be confused with cblocks, which
-are internal to CFiles, discussed below).
+written to a single BlockManager Block (not to be confused with CFile blocks,
+which are internal to CFiles, and discussed for the remainder of this document).
-A CFile is composed of a header, a variable number of cblocks, and a footer.
-There are a few different types of cblocks, each with a different format: data
-cblocks, nullable data cblocks, index cblocks, and dictionary cblocks.
+A CFile is composed of a header, a variable number of blocks, and a footer.
+There are a few different types of blocks, each with a different format: data
+blocks, nullable data blocks, index blocks, and dictionary blocks.
-Data cblocks consist of a sequence of consecutive values. Each value is assigned
-an ordinal index, or offset, which is unique within the CFile. Thus, a cblock
-can be identified within a CFile by the range of ordinal indexes it contains;
-for instance, the first three cblocks of a CFile might have the ordinal index
-ranges `[0, 55]`, `[56, 132]`, and `[133, 511]`.
+Data blocks consist of a sequence of consecutive values. Each value is assigned
+an ordinal index, or offset, which is unique within the CFile. Thus, a block can
+be identified within a CFile by the range of ordinal indexes it contains; for
+instance, the first three blocks of a CFile might have the ordinal index ranges
+`[0, 55]`, `[56, 132]`, and `[133, 511]`.
In order to support efficient seeks to an arbitrary ordinal index within the
CFile, a positional index may be included which contains a mapping of starting
-ordinal index to data cblock. See [CFile Index](#cfile-index) for more details.
+ordinal index to data block. See [CFile Index](#cfile-index) for more details.
For CFiles which contain data in sorted order (for example, a CFile containing
the Primary Key column), a value index may be created. The value index supports
@@ -47,7 +47,7 @@ efficiently seeking to an arbitrary value in the CFile. See
## File Format
-The CFile header, cblocks, and footer are written consecutively to the file
+The CFile header, blocks, and footer are written consecutively to the file
without padding.
### Header Layout
@@ -83,7 +83,7 @@ CFile format. At the same time, two sets of feature flags were introduced to the
footer in order to make more granular forwards compatibility possible in the
future. See commit [`f82cf6918c`] for details.
-### Data cblock Layout
+### Data Block Layout
| field | width (bytes) | notes |
| --- | --- | --- |
@@ -93,9 +93,9 @@ future. See commit [`f82cf6918c`] for details.
If the CFile is configured to use compression then the encoded data is
compressed before the checksum is appended.
-### Nullable Data cblock Layout
+### Nullable Data Block Layout
-Columns marked as nullable in the schema are stored in a nullable cblock, which
+Columns marked as nullable in the schema are stored in a nullable block, which
includes a bitmap which tracks which rows (ordinal offsets) are null and not
null.
@@ -125,11 +125,11 @@ exist, the reader can optionally validate them against the read data.
## Data Encodings
-cblock data is encoded before being stored. If compression is enabled, then the
-cblock is encoded first, and then compressed.
+Block data is encoded before being stored. If compression is enabled, then the
+block is encoded first, and then compressed.
-Data cblocks are best-effort size limited to `--cfile-default-block-size` bytes, at
-which point a new cblock will be added to the CFile.
+Data blocks are best-effort size limited to `--cfile-default-block-size` bytes, at
+which point a new block will be added to the CFile.
### Plain Encoding
@@ -138,7 +138,7 @@ natural representation.
Plain encoding for fixed-width (integer) types consists of the little-endian
values, followed by a trailer containing two little-endian `uint32`s: the count
-of values, and the ordinal position of the cblock.
+of values, and the ordinal position of the block.
Plain encoding for BINARY or STRING (variable-width) columns is laid out as
follows:
@@ -153,15 +153,14 @@ follows:
### Dictionary Encoding
[Dictionary encoding](dictionary-encoding) may be used for BINARY or STRING
-columns. All dictionary encoded cblocks in a CFile share the same dictionary.
-If the dictionary becomes full, subsequent cblocks in the CFile switch to plain
-encoding. The dictionary is stored as a plain encoded binary cblock, and the
-data codewords are stored as [bitshuffle encoded](#bitshuffle-encoding)
-`uint32`s.
+columns. All dictionary encoded blocks in a CFile share the same dictionary. If
+the dictionary becomes full, subsequent blocks in the CFile switch to plain
+encoding. The dictionary is stored as a plain encoded binary block, and the data
+codewords are stored as [bitshuffle encoded](#bitshuffle-encoding) `uint32`s.
### Prefix Encoding
-Currently used for `BINARY` and `STRING` data cblocks. This is based on the
+Currently used for `BINARY` and `STRING` data blocks. This is based on the
encoding used by LevelDB for its data blocks, more or less.
Starts with a header of four [Group Varint] encoded `uint32`s:
@@ -186,20 +185,20 @@ Periodically there will be a "restart point" which is necessary for faster
binary searching. At a "restart point", `shared_bytes` is 0 but otherwise the
encoding is the same.
-At the end of the cblock is a trailer with the offsets of the restart points:
+At the end of the block is a trailer with the offsets of the restart points:
| field | type |
| --- | --- |
| `restart_points` | `uint32[num_restarts]` |
| `num_restarts` | `uint32` |
-The restart points are offsets relative to the start of the cblock, including
-the header.
+The restart points are offsets relative to the start of the block, including the
+header.
### Run-length Encoding
Integer and bool types may be [run-length encoded](RLE), which has a simply
-layout: the number of values and ordinal position of the cblock as little-endian
+layout: the number of values and ordinal position of the block as little-endian
`uint32`s, followed by the run-length encoded data. The encoder will
automatically fall back to a bit-packed (literal) encoding if the data is not
conducive to run-length encoding.
@@ -207,7 +206,7 @@ conducive to run-length encoding.
### Bitshuffle Encoding
Integer types may be [bitshuffle](bitshuffle) encoded. Bitshuffle-encoded
-cblocks are automatically LZ4 compressed, so additional compression is not
+blocks are automatically LZ4 compressed, so additional compression is not
recommended.
### Group Varint Frame-of-Reference Encoding
@@ -225,8 +224,8 @@ Starts with a header of four [group-varint](group-varint) encoded `uint32`s:
| unused |
The ordinal position is the ordinal position of the first item in the file. For
-example, the first data cblock in the file has ordinal position 0. If that cblock
-had 400 values, then the second data cblock would have ordinal position
+example, the first data block in the file has ordinal position 0. If that block
+had 400 values, then the second data block would have ordinal position
400.
Followed by the actual data, each set of 4 integers using group-varint. The last
@@ -236,18 +235,18 @@ header.
## CFile Index
CFiles may optionally include a positional (ordinal) index and value index.
-Positional indexes are used to satisfy queries such as: "seek to the data cblock
+Positional indexes are used to satisfy queries such as: "seek to the data block
containing the Nth entry in this CFile". Value indexes are used to satisfy
-queries such as: "seek to the data cblock containing `123` in this CFile". Value
+queries such as: "seek to the data block containing `123` in this CFile". Value
indexes are only present in CFiles which contain data stored in sorted order
(e.g. the primary key column).
-Ordinal and value indices are organized as a [B-Tree] of index cblocks. As data
-cblocks are written, entries are appended to the end of a leaf index cblock.
-When a leaf index cblock reaches the configured index cblock size, it is added
-to another index cblock higher up the tree, and a new leaf is started. If the
-intermediate index cblock fills, it will start a new intermediate cblock and
-spill into an even higher-layer internal cblock.
+Ordinal and value indices are organized as a [B-Tree] of index blocks. As data
+blocks are written, entries are appended to the end of a leaf index block.
+When a leaf index block reaches the configured index block size, it is added
+to another index block higher up the tree, and a new leaf is started. If the
+intermediate index block fills, it will start a new intermediate block and
+spill into an even higher-layer internal block.
For example:
@@ -271,32 +270,32 @@ Note that this strategy doesn't result in a fully balanced B-tree, but instead
results in a 100% "fill factor" on all nodes in each level except for the last
one written.
-An index cblock is encoded similarly for both types of indexes:
+An index block is encoded similarly for both types of indexes:
```
-<key> <cblock offset> <cblock size>
-<key> <cblock offset> <cblock size>
+<key> <block offset> <block size>
+<key> <block offset> <block size>
...
key: vint64 for positional, otherwise varint-length-prefixed string
offset: vint64
- cblock size: vint32
+ block size: vint32
<offset to first key> (fixed32)
<offset to second key> (fixed32)
...
- These offsets are relative to the start of the cblock.
+ These offsets are relative to the start of the block.
<trailer>
A IndexBlockTrailerPB protobuf
<trailer length>
```
-Index cblocks are written using a [post-order traversal], and the index cblocks
-may intersperse with the data cblocks.
+Index blocks are written using a [post-order traversal], and the index blocks
+may intersperse with the data blocks.
-The trailer protobuf includes a field which designates whether the cblock is a
+The trailer protobuf includes a field which designates whether the block is a
leaf node or internal node of the B-Tree, allowing a reader to know whether the
-pointer is to another index cblock or to a data cblock.
+pointer is to another index block or to a data block.
## DeltaFile
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/cfile/cfile_reader.cc
----------------------------------------------------------------------
diff --git a/src/kudu/cfile/cfile_reader.cc b/src/kudu/cfile/cfile_reader.cc
index 1a4614f..19323ee 100644
--- a/src/kudu/cfile/cfile_reader.cc
+++ b/src/kudu/cfile/cfile_reader.cc
@@ -685,7 +685,7 @@ Status CFileIterator::SeekToOrdinal(rowid_t ord_idx) {
prepared_blocks_.push_back(b);
} else {
- // Otherwise, prepare a new cblock to scan.
+ // Otherwise, prepare a new block to scan.
RETURN_NOT_OK(PrepareForNewSeek());
if (PREDICT_FALSE(posidx_iter_ == nullptr)) {
return Status::NotSupported("no positional index in file");
@@ -940,7 +940,7 @@ Status CFileIterator::ReadCurrentDataBlock(const IndexTreeIterator &idx_iter,
}
io_stats_.cells_read += num_rows_in_block;
- io_stats_.cblocks_read++;
+ io_stats_.blocks_read++;
io_stats_.bytes_read += data_block.size();
prep_block->idx_in_block_ = 0;
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/common/iterator_stats.cc
----------------------------------------------------------------------
diff --git a/src/kudu/common/iterator_stats.cc b/src/kudu/common/iterator_stats.cc
index 42d2b57..878e9bb 100644
--- a/src/kudu/common/iterator_stats.cc
+++ b/src/kudu/common/iterator_stats.cc
@@ -29,18 +29,18 @@ using strings::Substitute;
IteratorStats::IteratorStats()
: cells_read(0),
bytes_read(0),
- cblocks_read(0) {
+ blocks_read(0) {
}
string IteratorStats::ToString() const {
- return Substitute("cells_read=$0 bytes_read=$1 cblocks_read=$2",
- cells_read, bytes_read, cblocks_read);
+ return Substitute("cells_read=$0 bytes_read=$1 blocks_read=$2",
+ cells_read, bytes_read, blocks_read);
}
IteratorStats& IteratorStats::operator+=(const IteratorStats& other) {
cells_read += other.cells_read;
bytes_read += other.bytes_read;
- cblocks_read += other.cblocks_read;
+ blocks_read += other.blocks_read;
DCheckNonNegative();
return *this;
}
@@ -48,7 +48,7 @@ IteratorStats& IteratorStats::operator+=(const IteratorStats& other) {
IteratorStats& IteratorStats::operator-=(const IteratorStats& other) {
cells_read -= other.cells_read;
bytes_read -= other.bytes_read;
- cblocks_read -= other.cblocks_read;
+ blocks_read -= other.blocks_read;
DCheckNonNegative();
return *this;
}
@@ -68,6 +68,6 @@ IteratorStats IteratorStats::operator-(const IteratorStats& other) {
void IteratorStats::DCheckNonNegative() const {
DCHECK_GE(cells_read, 0);
DCHECK_GE(bytes_read, 0);
- DCHECK_GE(cblocks_read, 0);
+ DCHECK_GE(blocks_read, 0);
}
} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/common/iterator_stats.h
----------------------------------------------------------------------
diff --git a/src/kudu/common/iterator_stats.h b/src/kudu/common/iterator_stats.h
index 542c87f..df0b69e 100644
--- a/src/kudu/common/iterator_stats.h
+++ b/src/kudu/common/iterator_stats.h
@@ -35,7 +35,7 @@ struct IteratorStats {
int64_t bytes_read;
// The number of CFile data blocks read from disk (or cache) by the iterator.
- int64_t cblocks_read;
+ int64_t blocks_read;
// Add statistics contained 'other' to this object (for each field
// in this object, increment it by the value of the equivalent field
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/common/wire_protocol.cc
----------------------------------------------------------------------
diff --git a/src/kudu/common/wire_protocol.cc b/src/kudu/common/wire_protocol.cc
index e4c8606..25b530a 100644
--- a/src/kudu/common/wire_protocol.cc
+++ b/src/kudu/common/wire_protocol.cc
@@ -725,12 +725,12 @@ static void CopyColumn(const RowBlock& block, int col_idx, int dst_col_idx, uint
faststring* indirect_data, const Schema* dst_schema, size_t row_stride,
size_t schema_byte_size, size_t column_offset) {
DCHECK(dst_schema);
- ColumnBlock cblock = block.column_block(col_idx);
+ ColumnBlock column_block = block.column_block(col_idx);
uint8_t* dst = dst_base + column_offset;
size_t offset_to_null_bitmap = schema_byte_size - column_offset;
- size_t cell_size = cblock.stride();
- const uint8_t* src = cblock.cell_ptr(0);
+ size_t cell_size = column_block.stride();
+ const uint8_t* src = column_block.cell_ptr(0);
BitmapIterator selected_row_iter(block.selection_vector()->bitmap(), block.nrows());
int run_size;
@@ -743,7 +743,7 @@ static void CopyColumn(const RowBlock& block, int col_idx, int dst_col_idx, uint
continue;
}
for (int i = 0; i < run_size; i++) {
- if (IS_NULLABLE && cblock.is_null(row_idx)) {
+ if (IS_NULLABLE && column_block.is_null(row_idx)) {
BitmapChange(dst + offset_to_null_bitmap, dst_col_idx, true);
} else if (IS_VARLEN) {
const Slice *slice = reinterpret_cast<const Slice *>(src);
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/tablet/cfile_set-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/cfile_set-test.cc b/src/kudu/tablet/cfile_set-test.cc
index 11e0f8b..a1c4316 100644
--- a/src/kudu/tablet/cfile_set-test.cc
+++ b/src/kudu/tablet/cfile_set-test.cc
@@ -227,10 +227,10 @@ TEST_F(TestCFileSet, TestPartiallyMaterialize) {
}
// Since we pushed down the block size, we expect to have read 100+ blocks of column 0
- ASSERT_GT(stats[0].cblocks_read, 100);
+ ASSERT_GT(stats[0].blocks_read, 100);
// Since we didn't ever materialize column 2, we shouldn't have read any data blocks.
- ASSERT_EQ(0, stats[2].cblocks_read);
+ ASSERT_EQ(0, stats[2].blocks_read);
// Column 0 and 1 skipped a lot of blocks, so should not have read all of the cells
// from either column.
@@ -317,9 +317,9 @@ TEST_F(TestCFileSet, TestRangeScan) {
vector<IteratorStats> stats;
iter->GetIteratorStats(&stats);
ASSERT_EQ(3, stats.size());
- EXPECT_EQ(1, stats[0].cblocks_read);
- EXPECT_EQ(1, stats[1].cblocks_read);
- EXPECT_EQ(1, stats[2].cblocks_read);
+ EXPECT_EQ(1, stats[0].blocks_read);
+ EXPECT_EQ(1, stats[1].blocks_read);
+ EXPECT_EQ(1, stats[2].blocks_read);
}
// Several other black-box tests for range scans. These are similar to
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/tablet/deltafile.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/deltafile.cc b/src/kudu/tablet/deltafile.cc
index 1d773f9..fbc58f2 100644
--- a/src/kudu/tablet/deltafile.cc
+++ b/src/kudu/tablet/deltafile.cc
@@ -107,7 +107,7 @@ DeltaFileWriter::DeltaFileWriter(unique_ptr<WritableBlock> block)
// The CFile value index is 'compressed' by truncating delta values to only
// contain the delta key. The entire deltakey is required in order to support
- // efficient seeks without deserializing the entire cblock. The generic value
+ // efficient seeks without deserializing the entire block. The generic value
// index optimization is disabled, since it could truncate portions of the
// deltakey.
//
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/tablet/tablet-pushdown-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/tablet-pushdown-test.cc b/src/kudu/tablet/tablet-pushdown-test.cc
index 2297afc..55fb5ef 100644
--- a/src/kudu/tablet/tablet-pushdown-test.cc
+++ b/src/kudu/tablet/tablet-pushdown-test.cc
@@ -160,7 +160,7 @@ class TabletPushdownTest : public KuduTabletTest,
vector<IteratorStats> stats;
iter->GetIteratorStats(&stats);
for (const IteratorStats& col_stats : stats) {
- EXPECT_EQ(expected_blocks_from_disk, col_stats.cblocks_read);
+ EXPECT_EQ(expected_blocks_from_disk, col_stats.blocks_read);
EXPECT_EQ(expected_rows_from_disk, col_stats.cells_read);
}
}
@@ -257,7 +257,7 @@ class TabletSparsePushdownTest : public KuduTabletTest {
};
// The is a regression test for KUDU-2231, which fixed a CFileReader bug causing
-// cblocks to be repeatedly materialized when a scan had predicates on other
+// blocks to be repeatedly materialized when a scan had predicates on other
// columns which caused sections of a column to be skipped. The setup for this
// test creates a dataset and predicate which only match every-other batch of
// 100 rows of the 'val' column.
@@ -283,8 +283,8 @@ TEST_F(TabletSparsePushdownTest, Kudu2231) {
iter->GetIteratorStats(&stats);
ASSERT_EQ(2, stats.size());
- EXPECT_EQ(1, stats[0].cblocks_read);
- EXPECT_EQ(1, stats[1].cblocks_read);
+ EXPECT_EQ(1, stats[0].blocks_read);
+ EXPECT_EQ(1, stats[1].blocks_read);
EXPECT_EQ(400, stats[0].cells_read);
EXPECT_EQ(400, stats[1].cells_read);
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/src/kudu/tserver/tserver_path_handlers.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tserver/tserver_path_handlers.cc b/src/kudu/tserver/tserver_path_handlers.cc
index 19b8bb6..ae4aa51 100644
--- a/src/kudu/tserver/tserver_path_handlers.cc
+++ b/src/kudu/tserver/tserver_path_handlers.cc
@@ -555,11 +555,11 @@ void IteratorStatsToJson(const ScanDescriptor& scan, EasyJson* json) {
row["bytes_read"] = HumanReadableNumBytes::ToString(stats.bytes_read);
row["cells_read"] = HumanReadableInt::ToString(stats.cells_read);
- row["cblocks_read"] = HumanReadableInt::ToString(stats.cblocks_read);
+ row["blocks_read"] = HumanReadableInt::ToString(stats.blocks_read);
row["bytes_read_title"] = stats.bytes_read;
row["cells_read_title"] = stats.cells_read;
- row["cblocks_read_title"] = stats.cblocks_read;
+ row["blocks_read_title"] = stats.blocks_read;
};
IteratorStats total_stats;
http://git-wip-us.apache.org/repos/asf/kudu/blob/9941b56b/www/scans.mustache
----------------------------------------------------------------------
diff --git a/www/scans.mustache b/www/scans.mustache
index 3c6edad..8f2936b 100644
--- a/www/scans.mustache
+++ b/www/scans.mustache
@@ -50,7 +50,7 @@ under the License.
<th>column</th>
<th title="cells read from the column (disk or cache), exclusive of the MRS">cells read</th>
<th title="bytes read from the column (disk or cache), exclusive of the MRS">bytes read</th>
- <th title="CFile data blocks read from the column (disk or cache)">cblocks read</th>
+ <th title="CFile data blocks read from the column (disk or cache)">blocks read</th>
</tr>
</thead>
<tbody>
@@ -59,7 +59,7 @@ under the License.
<td>{{column}}</td>
<td title="{{cells_read_title}}">{{cells_read}}</td>
<td title="{{bytes_read_title}}">{{bytes_read}}</td>
- <td title="{{cblocks_read_title}}">{{cblocks_read}}</td>
+ <td title="{{blocks_read_title}}">{{blocks_read}}</td>
</tr>
{{/stats}}
</tbody>