You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2019/10/02 04:11:17 UTC
[kudu] branch master updated (8ddeda6 -> f73671a)
This is an automated email from the ASF dual-hosted git repository.
adar pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 8ddeda6 [clock] modernize clock-related classes
new e7d9d02 [docs] Fix code block formatting
new 62c2fdd cfile: track memory for CFileReaders and BloomFileReaders
new 09ffff4 TestMiniKuduCluster: deflake testHiveMetastoreIntegration
new 3f6d3c4 AsyncKuduClient: improve debug-level getTableKeyRanges logging
new f73671a TestKuduPartitioner: deflake testBuildTimeout
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/design-docs/tablet.md | 10 ++++++++--
.../org/apache/kudu/client/AsyncKuduClient.java | 3 ++-
.../apache/kudu/client/TestKuduPartitioner.java | 4 +++-
.../org/apache/kudu/test/TestMiniKuduCluster.java | 2 +-
src/kudu/tablet/cfile_set-test.cc | 15 ++++++++++-----
src/kudu/tablet/cfile_set.cc | 22 +++++++++++++---------
src/kudu/tablet/cfile_set.h | 9 ++++++---
src/kudu/tablet/diskrowset.cc | 6 ++++--
src/kudu/tablet/tablet_mem_trackers.h | 8 +++++++-
9 files changed, 54 insertions(+), 25 deletions(-)
[kudu] 03/05: TestMiniKuduCluster: deflake
testHiveMetastoreIntegration
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 09ffff48d4122f4ea44fa4ff841cc089a2fc1aea
Author: Adar Dembo <ad...@cloudera.com>
AuthorDate: Tue Oct 1 12:16:30 2019 -0700
TestMiniKuduCluster: deflake testHiveMetastoreIntegration
50 seconds for the entire test isn't enough considering just the HMS itself
is given up to 60s to start (see mini_hms.cc). So let's double it.
Change-Id: I53fb1fa826e3c5c5d9d032fa88bf584300aeedfe
Reviewed-on: http://gerrit.cloudera.org:8080/14335
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Hao Hao <ha...@cloudera.com>
Tested-by: Kudu Jenkins
---
.../src/test/java/org/apache/kudu/test/TestMiniKuduCluster.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/java/kudu-test-utils/src/test/java/org/apache/kudu/test/TestMiniKuduCluster.java b/java/kudu-test-utils/src/test/java/org/apache/kudu/test/TestMiniKuduCluster.java
index 68b3933..63cf68b 100644
--- a/java/kudu-test-utils/src/test/java/org/apache/kudu/test/TestMiniKuduCluster.java
+++ b/java/kudu-test-utils/src/test/java/org/apache/kudu/test/TestMiniKuduCluster.java
@@ -97,7 +97,7 @@ public class TestMiniKuduCluster {
}
}
- @Test(timeout = 50000)
+ @Test(timeout = 100000)
public void testHiveMetastoreIntegration() throws Exception {
try (MiniKuduCluster cluster = new MiniKuduCluster.MiniKuduClusterBuilder()
.numMasterServers(NUM_MASTERS)
[kudu] 05/05: TestKuduPartitioner: deflake testBuildTimeout
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit f73671a5a619d543dcda61179388c496a7ca1352
Author: Adar Dembo <ad...@cloudera.com>
AuthorDate: Mon Sep 30 22:44:54 2019 -0700
TestKuduPartitioner: deflake testBuildTimeout
I saw a RELEASE build where the time taken by KuduPartitionerBuilder.build()
exceeded 110% of the timeout value. This was anticipated in code review[1],
but instead of removing this part of the test altogether, let's just make it
more lenient.
1. https://gerrit.cloudera.org/c/12275/6/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java#192
Change-Id: I9e734eb8321b6633d1999e2dd031d355ce299b86
Reviewed-on: http://gerrit.cloudera.org:8080/14331
Tested-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Grant Henke <gr...@apache.org>
---
.../src/test/java/org/apache/kudu/client/TestKuduPartitioner.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java
index 743926c..400c8e6 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java
@@ -184,7 +184,9 @@ public class TestKuduPartitioner {
assertTrue(ex.getMessage().startsWith("cannot complete before timeout"));
}
long elapsed = System.currentTimeMillis() - now;
- assertTrue(elapsed <= timeoutMs * 1.1); // Add 10% to avoid flakiness.
+ long upperBound = timeoutMs * 2; // Add 100% to avoid flakiness.
+ assertTrue(String.format("Elapsed time %d exceeded upper bound %d", elapsed, upperBound),
+ elapsed <= upperBound);
}
@Test
[kudu] 01/05: [docs] Fix code block formatting
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e7d9d02479d5e3801b711164a157c51a2c9109c5
Author: lingbin <li...@gmail.com>
AuthorDate: Mon Sep 30 13:25:38 2019 +0800
[docs] Fix code block formatting
Change-Id: Ibc54ba74ee58e1c601bf7e60a3a51c6d48659345
Reviewed-on: http://gerrit.cloudera.org:8080/14320
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
docs/design-docs/tablet.md | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/docs/design-docs/tablet.md b/docs/design-docs/tablet.md
index fb2d0fe..6e581a4 100644
--- a/docs/design-docs/tablet.md
+++ b/docs/design-docs/tablet.md
@@ -453,8 +453,8 @@ number of REDO delta files.
```
Major delta compactions satisfy delta compaction goals 1 and 2, but cost more
-than than minor delta compactions since they must read and re-write the base
-data, which is typically larger than the delta data.
+than minor delta compactions since they must read and re-write the base data,
+which is typically larger than the delta data.
A major REDO delta compaction may be performed against any subset of the columns
in a DiskRowSet -- if only a single column has received a significant number of updates,
@@ -647,11 +647,16 @@ Additionally, if the key is not needed in the query results, the query plan
need not consult the key except perhaps to determine scan boundaries.
As an example, consider the query:
+
+```
> SELECT SUM(cpu_usage) FROM timeseries WHERE machine = 'foo.cloudera.com'
AND unix_time BETWEEN 1349658729 AND 1352250720;
... given a composite primary key (host, unix_time)
+```
This may be evaluated in Kudu with the following pseudo-code:
+
+```
sum = 0
foreach RowSet:
start_rowid = rowset.lookup_key(1349658729)
@@ -662,6 +667,7 @@ This may be evaluated in Kudu with the following pseudo-code:
while remaining > 0:
block = iter.fetch_upto(remaining)
sum += sum(block)
+```
The fetching of blocks can be done very efficiently since the application
of any potential mutations can simply index into the block and replace
[kudu] 02/05: cfile: track memory for CFileReaders and
BloomFileReaders
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 62c2fddcdc65f2092ee335ac1a35288dd3f7e4a5
Author: Andrew Wong <aw...@apache.org>
AuthorDate: Tue Oct 1 13:30:05 2019 -0700
cfile: track memory for CFileReaders and BloomFileReaders
This lumps a tablet's CFileReader tracked memory and BloomFileReader
tracked memory into the ids "CFileReaders" and "BloomFileReaders"
respectively.
I test this locally and verified via the web UI that I could see
CFileReader and BloomFileReader entries once they had been flushed.
Change-Id: Ic62c89079edffb41d1f5b40d7b699c525fa447a7
Reviewed-on: http://gerrit.cloudera.org:8080/14322
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
src/kudu/tablet/cfile_set-test.cc | 15 ++++++++++-----
src/kudu/tablet/cfile_set.cc | 22 +++++++++++++---------
src/kudu/tablet/cfile_set.h | 9 ++++++---
src/kudu/tablet/diskrowset.cc | 6 ++++--
src/kudu/tablet/tablet_mem_trackers.h | 8 +++++++-
5 files changed, 40 insertions(+), 20 deletions(-)
diff --git a/src/kudu/tablet/cfile_set-test.cc b/src/kudu/tablet/cfile_set-test.cc
index e3fd517..c074e9d 100644
--- a/src/kudu/tablet/cfile_set-test.cc
+++ b/src/kudu/tablet/cfile_set-test.cc
@@ -278,7 +278,8 @@ TEST_F(TestCFileSet, TestPartiallyMaterialize) {
WriteTestRowSet(kNumRows);
shared_ptr<CFileSet> fileset;
- ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), nullptr, &fileset));
+ ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), MemTracker::GetRootTracker(),
+ nullptr, &fileset));
unique_ptr<CFileSet::Iterator> iter(fileset->NewIterator(&schema_, nullptr));
ASSERT_OK(iter->Init(nullptr));
@@ -358,7 +359,8 @@ TEST_F(TestCFileSet, TestIteratePartialSchema) {
WriteTestRowSet(kNumRows);
shared_ptr<CFileSet> fileset;
- ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), nullptr, &fileset));
+ ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), MemTracker::GetRootTracker(),
+ nullptr, &fileset));
Schema new_schema;
ASSERT_OK(schema_.CreateProjectionByNames({ "c0", "c2" }, &new_schema));
@@ -391,7 +393,8 @@ TEST_F(TestCFileSet, TestRangeScan) {
WriteTestRowSet(kNumRows);
shared_ptr<CFileSet> fileset;
- ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), nullptr, &fileset));
+ ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), MemTracker::GetRootTracker(),
+ nullptr, &fileset));
// Create iterator.
unique_ptr<CFileSet::Iterator> cfile_iter(fileset->NewIterator(&schema_, nullptr));
@@ -445,7 +448,8 @@ TEST_F(TestCFileSet, TestRangePredicates2) {
WriteTestRowSet(kNumRows);
shared_ptr<CFileSet> fileset;
- ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), nullptr, &fileset));
+ ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), MemTracker::GetRootTracker(),
+ nullptr, &fileset));
// Range scan where rows match on both ends
DoTestRangeScan(fileset, 2000, 2010);
@@ -501,7 +505,8 @@ TEST_F(TestCFileSet, TestBloomFilterPredicates) {
&ret1_contain, &ret1_exclude, &ret2_contain, &ret2_exclude);
shared_ptr<CFileSet> fileset;
- ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), nullptr, &fileset));
+ ASSERT_OK(CFileSet::Open(rowset_meta_, MemTracker::GetRootTracker(), MemTracker::GetRootTracker(),
+ nullptr, &fileset));
vector<ColumnPredicate::BloomFilterInner> bfs;
// BloomFilter of column 0 contain.
diff --git a/src/kudu/tablet/cfile_set.cc b/src/kudu/tablet/cfile_set.cc
index 52a892b..e1b7881 100644
--- a/src/kudu/tablet/cfile_set.cc
+++ b/src/kudu/tablet/cfile_set.cc
@@ -88,7 +88,7 @@ using strings::Substitute;
////////////////////////////////////////////////////////////
static Status OpenReader(FsManager* fs,
- shared_ptr<MemTracker> parent_mem_tracker,
+ shared_ptr<MemTracker> cfile_reader_tracker,
const BlockId& block_id,
const IOContext* io_context,
unique_ptr<CFileReader>* new_reader) {
@@ -96,7 +96,7 @@ static Status OpenReader(FsManager* fs,
RETURN_NOT_OK(fs->OpenBlock(block_id, &block));
ReaderOptions opts;
- opts.parent_mem_tracker = std::move(parent_mem_tracker);
+ opts.parent_mem_tracker = std::move(cfile_reader_tracker);
opts.io_context = io_context;
return CFileReader::OpenNoInit(std::move(block),
std::move(opts),
@@ -108,20 +108,24 @@ static Status OpenReader(FsManager* fs,
////////////////////////////////////////////////////////////
CFileSet::CFileSet(shared_ptr<RowSetMetadata> rowset_metadata,
- shared_ptr<MemTracker> parent_mem_tracker)
+ shared_ptr<MemTracker> bloomfile_tracker,
+ shared_ptr<MemTracker> cfile_reader_tracker)
: rowset_metadata_(std::move(rowset_metadata)),
- parent_mem_tracker_(std::move(parent_mem_tracker)) {
+ bloomfile_tracker_(std::move(bloomfile_tracker)),
+ cfile_reader_tracker_(std::move(cfile_reader_tracker)) {
}
CFileSet::~CFileSet() {
}
Status CFileSet::Open(shared_ptr<RowSetMetadata> rowset_metadata,
- shared_ptr<MemTracker> parent_mem_tracker,
+ shared_ptr<MemTracker> bloomfile_tracker,
+ shared_ptr<MemTracker> cfile_reader_tracker,
const IOContext* io_context,
shared_ptr<CFileSet>* cfile_set) {
shared_ptr<CFileSet> cfs(new CFileSet(std::move(rowset_metadata),
- std::move(parent_mem_tracker)));
+ std::move(bloomfile_tracker),
+ std::move(cfile_reader_tracker)));
RETURN_NOT_OK(cfs->DoOpen(io_context));
cfile_set->swap(cfs);
@@ -140,7 +144,7 @@ Status CFileSet::DoOpen(const IOContext* io_context) {
unique_ptr<CFileReader> reader;
RETURN_NOT_OK(OpenReader(rowset_metadata_->fs_manager(),
- parent_mem_tracker_,
+ cfile_reader_tracker_,
rowset_metadata_->column_data_block_for_col_id(col_id),
io_context,
&reader));
@@ -152,7 +156,7 @@ Status CFileSet::DoOpen(const IOContext* io_context) {
if (rowset_metadata_->has_adhoc_index_block()) {
RETURN_NOT_OK(OpenReader(rowset_metadata_->fs_manager(),
- parent_mem_tracker_,
+ cfile_reader_tracker_,
rowset_metadata_->adhoc_index_block(),
io_context,
&ad_hoc_idx_reader_));
@@ -183,8 +187,8 @@ Status CFileSet::OpenBloomReader(const IOContext* io_context) {
RETURN_NOT_OK(fs->OpenBlock(rowset_metadata_->bloom_block(), &block));
ReaderOptions opts;
- opts.parent_mem_tracker = parent_mem_tracker_;
opts.io_context = io_context;
+ opts.parent_mem_tracker = bloomfile_tracker_;
Status s = BloomFileReader::OpenNoInit(std::move(block),
std::move(opts),
&bloom_reader_);
diff --git a/src/kudu/tablet/cfile_set.h b/src/kudu/tablet/cfile_set.h
index 4721e95..a34dd21 100644
--- a/src/kudu/tablet/cfile_set.h
+++ b/src/kudu/tablet/cfile_set.h
@@ -74,7 +74,8 @@ class CFileSet : public std::enable_shared_from_this<CFileSet> {
class Iterator;
static Status Open(std::shared_ptr<RowSetMetadata> rowset_metadata,
- std::shared_ptr<MemTracker> parent_mem_tracker,
+ std::shared_ptr<MemTracker> bloomfile_tracker,
+ std::shared_ptr<MemTracker> cfile_reader_tracker,
const fs::IOContext* io_context,
std::shared_ptr<CFileSet>* cfile_set);
@@ -133,7 +134,8 @@ class CFileSet : public std::enable_shared_from_this<CFileSet> {
DISALLOW_COPY_AND_ASSIGN(CFileSet);
CFileSet(std::shared_ptr<RowSetMetadata> rowset_metadata,
- std::shared_ptr<MemTracker> parent_mem_tracker);
+ std::shared_ptr<MemTracker> bloomfile_tracker,
+ std::shared_ptr<MemTracker> cfile_reader_tracker);
Status DoOpen(const fs::IOContext* io_context);
Status OpenBloomReader(const fs::IOContext* io_context);
@@ -153,7 +155,8 @@ class CFileSet : public std::enable_shared_from_this<CFileSet> {
const Schema &tablet_schema() const { return rowset_metadata_->tablet_schema(); }
std::shared_ptr<RowSetMetadata> rowset_metadata_;
- std::shared_ptr<MemTracker> parent_mem_tracker_;
+ std::shared_ptr<MemTracker> bloomfile_tracker_;
+ std::shared_ptr<MemTracker> cfile_reader_tracker_;
std::string min_encoded_key_;
std::string max_encoded_key_;
diff --git a/src/kudu/tablet/diskrowset.cc b/src/kudu/tablet/diskrowset.cc
index b2395a9..f4dd37d 100644
--- a/src/kudu/tablet/diskrowset.cc
+++ b/src/kudu/tablet/diskrowset.cc
@@ -535,7 +535,8 @@ DiskRowSet::DiskRowSet(shared_ptr<RowSetMetadata> rowset_metadata,
Status DiskRowSet::Open(const IOContext* io_context) {
TRACE_EVENT0("tablet", "DiskRowSet::Open");
RETURN_NOT_OK(CFileSet::Open(rowset_metadata_,
- mem_trackers_.tablet_tracker,
+ mem_trackers_.bloomfile_tracker,
+ mem_trackers_.cfile_reader_tracker,
io_context,
&base_data_));
@@ -607,7 +608,8 @@ Status DiskRowSet::MajorCompactDeltaStoresWithColumnIds(const vector<ColumnId>&
// appropriate blocks to match the update.
shared_ptr<CFileSet> new_base;
RETURN_NOT_OK(CFileSet::Open(rowset_metadata_,
- mem_trackers_.tablet_tracker,
+ mem_trackers_.bloomfile_tracker,
+ mem_trackers_.cfile_reader_tracker,
io_context,
&new_base));
{
diff --git a/src/kudu/tablet/tablet_mem_trackers.h b/src/kudu/tablet/tablet_mem_trackers.h
index 4882318..405b4cb 100644
--- a/src/kudu/tablet/tablet_mem_trackers.h
+++ b/src/kudu/tablet/tablet_mem_trackers.h
@@ -30,6 +30,8 @@ struct TabletMemTrackers {
// Intended for unit tests where the tracker hierarchy doesn't matter.
TabletMemTrackers()
: tablet_tracker(MemTracker::GetRootTracker()),
+ bloomfile_tracker(MemTracker::GetRootTracker()),
+ cfile_reader_tracker(MemTracker::GetRootTracker()),
dms_tracker(MemTracker::GetRootTracker()) {
}
@@ -38,13 +40,17 @@ struct TabletMemTrackers {
: tablet_tracker(MemTracker::CreateTracker(
-1,
strings::Substitute("tablet-$0", tablet_id),
- parent_mem_tracker)),
+ std::move(parent_mem_tracker))),
+ bloomfile_tracker(MemTracker::CreateTracker(-1, "BloomFileReaders", tablet_tracker)),
+ cfile_reader_tracker(MemTracker::CreateTracker(-1, "CFileReaders", tablet_tracker)),
dms_tracker(MemTracker::CreateTracker(-1, "DeltaMemStores", tablet_tracker)) {
}
std::shared_ptr<MemTracker> tablet_tracker;
// All of the below are children of tablet_tracker;
+ std::shared_ptr<MemTracker> bloomfile_tracker;
+ std::shared_ptr<MemTracker> cfile_reader_tracker;
std::shared_ptr<MemTracker> dms_tracker;
};
[kudu] 04/05: AsyncKuduClient: improve debug-level
getTableKeyRanges logging
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3f6d3c46e8ca4cf5898eec50d61dbe91caa170e5
Author: Adar Dembo <ad...@cloudera.com>
AuthorDate: Tue Oct 1 14:01:00 2019 -0700
AsyncKuduClient: improve debug-level getTableKeyRanges logging
We've been seeing the occasional TestSplitKeyRange failure that's hard to
debug because the test makes multiple getTableKeyRanges() calls and it's not
clear to which call each line of output applies.
Change-Id: I5efb5b04db725741f20ec6b1b64849b029ce6db3
Reviewed-on: http://gerrit.cloudera.org:8080/14336
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
.../src/main/java/org/apache/kudu/client/AsyncKuduClient.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
index ba3824a..8f5050b 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
@@ -2006,13 +2006,14 @@ public class AsyncKuduClient implements AutoCloseable {
fakeRpc)
.addCallbackDeferring(resp -> {
final List<KeyRange> ranges = Lists.newArrayList();
+ LOG.debug("Key ranges for {}", table.getName());
for (Common.KeyRangePB pb : resp.getKeyRanges()) {
KeyRange newRange = new KeyRange(tablet,
pb.getStartPrimaryKey().toByteArray(),
pb.getStopPrimaryKey().toByteArray(),
pb.getSizeBytesEstimates());
ranges.add(newRange);
- LOG.debug("Add key range {}", newRange);
+ LOG.debug(newRange.toString());
}
return Deferred.fromResult(ranges);
}));