You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by al...@apache.org on 2019/10/11 06:15:20 UTC
[kudu] branch master updated (c29ee55 -> 6f49b2d)
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from c29ee55 webserver: port fix for IMPALA-9001 to Kudu
new b6e4672 [tablet] remove used method in Mutation
new 25d9284 [util] KUDU-2968 Fix RleDecoder::GetNextRun() in case of literal runs
new 6f49b2d [docs] a small update on RELEASING.adoc
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
RELEASING.adoc | 14 ++++++++------
src/kudu/tablet/mutation.h | 6 +-----
src/kudu/util/rle-encoding.h | 12 +++++++++---
src/kudu/util/rle-test.cc | 43 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 61 insertions(+), 14 deletions(-)
[kudu] 03/03: [docs] a small update on RELEASING.adoc
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 6f49b2dc076603d83d4ce2abaab47eab960b47bd
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Thu Oct 10 10:36:31 2019 -0700
[docs] a small update on RELEASING.adoc
Minor clarification on the username to use for logging in at
gerrit.cloudera.org to update gerrit mirroring configuration. Also,
fixed formatting of the numbered sub-list and added an item to verify
that gerrit has been restarted successfully.
Change-Id: I3046e24b63b1d70a0e5f4d5d28b5335a26255137
Reviewed-on: http://gerrit.cloudera.org:8080/14411
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
---
RELEASING.adoc | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/RELEASING.adoc b/RELEASING.adoc
index 6f19b46..a09a409 100644
--- a/RELEASING.adoc
+++ b/RELEASING.adoc
@@ -67,17 +67,19 @@ in `master`.
have to perform this step because SSH access is behind a firewall. The steps
are as follows:
1. Ensure your public SSH key is in `~gerrit/.ssh/authorized_keys` on gerrit.cloudera.org
- 2. From behind the firewall, `ssh gerrit.cloudera.org` to log in.
- 3. Change to the gerrit user, `sudo su gerrit`
- 4. Back up the existing replication configuration file by executing
+ 2. From behind the firewall, `ssh gerrit@gerrit.cloudera.org` to log in.
+ 3. Back up the existing replication configuration file by executing
`cp ~/etc/replication.config ~/etc/replication.config.bak.`date '+%Y%m%d.%H%M%S'``
- 5. Edit `etc/replication.config` to add a line for the new branch, such as `branch-1.x.y`
- 6. Send email to the dev lists for Kudu and Impala (dev@kudu.apache.org and
+ 4. Edit `etc/replication.config` to add a line for the new branch, such as `branch-1.x.y`
+ 5. Send email to the dev lists for Kudu and Impala (dev@kudu.apache.org and
dev@impala.apache.org) indicating that you are going to restart Gerrit
(link:https://s.apache.org/2Wj7[example]). It is best to do the restart at
some time of day when you don't expect many people to be using the system,
since Gerrit can take a few minutes to restart.
- 7: Restart Gerrit, `~/bin/gerrit.sh restart`
+ 6. Restart Gerrit: `~/bin/gerrit.sh restart`
+ 7. Make sure Gerrit has been successfully restarted: after a few minutes,
+ try to open the link:https://gerrit.cloudera.org/#/admin/projects[project list]
+ page in your favorite browser.
. As needed, patches can be cherry-picked to the new branch.
[kudu] 02/03: [util] KUDU-2968 Fix RleDecoder::GetNextRun() in case
of literal runs
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 25d92848ad7dc0852b23b9e967aa695005b8b374
Author: Bankim Bhavsar <ba...@cloudera.com>
AuthorDate: Tue Oct 8 18:06:23 2019 -0700
[util] KUDU-2968 Fix RleDecoder::GetNextRun() in case of literal runs
RLE encoding may encode "literally" when it doesn't find sufficient
repeated values.
When the number of values are not multiple of 8, literal count is
rounded up to multiple of 8. After reading the last value,
another attempt is made to read the value in GetNextRun() method.
This is beyond the max_bytes in the BitReader which returns false
and causes DCHECK() failure in GetNextRun() method.
For further details see JIRA KUDU-2968.
Fix is to check for the return value of BitReader::_GetValue()
and return without modifying further internal state of RleDecoder.
Similarly if an attempt is made by the caller of
RleDecoder::GetNextRun() to read after having read the last value
then BitReader::GetValue() would also return false again
leading to DCHECK() failure.
Tests:
- Added a test case for various int data types
that inserts wide range of non-repeated values
including non-multiple-of-8 and verfies GetNextRun() method.
Change-Id: If1703efd5307c66e282a2d2d7600dc4658c252df
Reviewed-on: http://gerrit.cloudera.org:8080/14405
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
src/kudu/util/rle-encoding.h | 12 +++++++++---
src/kudu/util/rle-test.cc | 43 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/src/kudu/util/rle-encoding.h b/src/kudu/util/rle-encoding.h
index 5b16fd9..f3cf2c1 100644
--- a/src/kudu/util/rle-encoding.h
+++ b/src/kudu/util/rle-encoding.h
@@ -318,8 +318,11 @@ inline size_t RleDecoder<T>::GetNextRun(T* val, size_t max_run) {
} else {
DCHECK(literal_count_ > 0);
if (ret == 0) {
- bool has_more = bit_reader_.GetValue(bit_width_, val);
- DCHECK(has_more);
+ bool result = bit_reader_.GetValue(bit_width_, val);
+ if (PREDICT_FALSE(!result)) {
+ return ret;
+ }
+
literal_count_--;
ret++;
rem--;
@@ -327,7 +330,10 @@ inline size_t RleDecoder<T>::GetNextRun(T* val, size_t max_run) {
while (literal_count_ > 0) {
bool result = bit_reader_.GetValue(bit_width_, ¤t_value_);
- DCHECK(result);
+ if (PREDICT_FALSE(!result)) {
+ return ret;
+ }
+
if (current_value_ != *val || rem == 0) {
bit_reader_.Rewind(bit_width_);
return ret;
diff --git a/src/kudu/util/rle-test.cc b/src/kudu/util/rle-test.cc
index 20d0a27..536c69b 100644
--- a/src/kudu/util/rle-test.cc
+++ b/src/kudu/util/rle-test.cc
@@ -543,4 +543,47 @@ TEST_F(TestRle, TestSkip) {
encoder.Flush();
}
+
+// RLE encoding groups values and decides whether to run-length encode or simply bit-pack
+// (literal encoding). This test verifies correctness of the RLE decoding when literal
+// encoding is used irrespective of the size of the group and the number of values encoded.
+template <typename IntType>
+class TestRleLiteralGetNextRun : public KuduTest {
+ public:
+ void RunTest() {
+ const auto num_bytes_per_val = sizeof(IntType);
+ const auto bit_width = 8 * num_bytes_per_val;
+
+ // Test with number of values that are not necessarily multiple of the group size (8).
+ auto max_num_vals = std::min<uint64_t>(1024, std::numeric_limits<IntType>::max());
+ for (auto num_vals = 1; num_vals <= max_num_vals; num_vals++) {
+ faststring buffer(num_vals * num_bytes_per_val);
+ RleEncoder<IntType> encoder(&buffer, bit_width);
+
+ // Use non-repeated pattern of integers so that literal encoding is used.
+ for (auto i = 0; i < num_vals; i++) {
+ encoder.Put(i, 1);
+ }
+ encoder.Flush();
+
+ RleDecoder<IntType> decoder(buffer.data(), encoder.len(), bit_width);
+ IntType val = 0;
+ for (auto i = 0; i < num_vals; i++) {
+ size_t len = decoder.GetNextRun(&val, num_vals);
+ ASSERT_EQ(1, len);
+ ASSERT_EQ(i, val);
+ }
+ // Read one beyond to verify no value is returned.
+ ASSERT_EQ(0, decoder.GetNextRun(&val, num_vals));
+ }
+ }
+};
+
+typedef ::testing::Types<int8_t, int16_t, int32_t, int64_t> IntDataTypes;
+TYPED_TEST_CASE(TestRleLiteralGetNextRun, IntDataTypes);
+
+TYPED_TEST(TestRleLiteralGetNextRun, RleGetNextRunIntDataTypes) {
+ this->RunTest();
+}
+
} // namespace kudu
[kudu] 01/03: [tablet] remove used method in Mutation
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit b6e4672092292acf7316a700c71fad3bb54ffa77
Author: lingbin <li...@gmail.com>
AuthorDate: Thu Oct 10 22:28:36 2019 +0800
[tablet] remove used method in Mutation
Change-Id: Ic3c7cded06ce7fead5f56371166e3908d16efa7a
Reviewed-on: http://gerrit.cloudera.org:8080/14409
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
src/kudu/tablet/mutation.h | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/src/kudu/tablet/mutation.h b/src/kudu/tablet/mutation.h
index 9071665..5a28c13 100644
--- a/src/kudu/tablet/mutation.h
+++ b/src/kudu/tablet/mutation.h
@@ -94,17 +94,13 @@ class Mutation {
*list = this;
}
- // O(n) algorithm to reverse the order of a linked list of
- // mutations.
+ // O(n) algorithm to reverse the order of a linked list of mutations.
static void ReverseMutationList(Mutation** list);
private:
friend class MSRow;
friend class MemRowSet;
- template<bool ATOMIC>
- void DoAppendToList(Mutation **list);
-
// The transaction ID which made this mutation. If this transaction is not
// committed in the snapshot of the reader, this mutation should be ignored.
Timestamp timestamp_;