You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by gr...@apache.org on 2018/04/30 21:10:16 UTC
[1/6] kudu git commit: KUDU-2422. Fix TSAN hangs in Subprocess::Start
Repository: kudu
Updated Branches:
refs/heads/master eef20da65 -> 69fb524c4
KUDU-2422. Fix TSAN hangs in Subprocess::Start
This pulls in an upstream patch from LLVM to fix an occasional hang when
starting processes in TSAN builds[1]
Tested with:
KUDU_ALLOW_SLOW_TESTS=0 ./build-support/dist_test.py loop -n 1000 \
./build/latest/bin/subprocess-test --gtest_filter=\*CurrentDir \
--gtest_repeat=1000 --stress-cpu-threads=3
Without the patch, two of the tasks hung in Subprocess::Start and
failed. With the patch, all succeeded.
[1] https://github.com/google/sanitizers/issues/945
Change-Id: I9426b28c8a925b6d59ec4241aa12007a07f6ec70
Reviewed-on: http://gerrit.cloudera.org:8080/10240
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/60acdb99
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/60acdb99
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/60acdb99
Branch: refs/heads/master
Commit: 60acdb99667f112894365ae24b65c247ed950f4a
Parents: eef20da
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Apr 30 08:57:37 2018 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Apr 30 18:12:44 2018 +0000
----------------------------------------------------------------------
thirdparty/download-thirdparty.sh | 5 +-
...race-switching-after-multithreaded-for.patch | 53 ++++++++++++++++++++
2 files changed, 56 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/60acdb99/thirdparty/download-thirdparty.sh
----------------------------------------------------------------------
diff --git a/thirdparty/download-thirdparty.sh b/thirdparty/download-thirdparty.sh
index ec5fb81..758a189 100755
--- a/thirdparty/download-thirdparty.sh
+++ b/thirdparty/download-thirdparty.sh
@@ -284,7 +284,7 @@ fetch_and_patch \
$PYTHON_SOURCE \
$PYTHON_PATCHLEVEL
-LLVM_PATCHLEVEL=1
+LLVM_PATCHLEVEL=2
fetch_and_patch \
llvm-${LLVM_VERSION}-iwyu-${IWYU_VERSION}.src.tar.gz \
$LLVM_SOURCE \
@@ -293,7 +293,8 @@ fetch_and_patch \
"patch -p1 < $TP_DIR/patches/llvm-add-iwyu.patch" \
"patch -p1 < $TP_DIR/patches/llvm-iwyu-nocurses.patch" \
"patch -p1 < $TP_DIR/patches/llvm-iwyu-include-picker.patch" \
- "patch -d tools/clang/tools/include-what-you-use -p1 < $TP_DIR/patches/llvm-iwyu-llvm-6-compat.patch"
+ "patch -d tools/clang/tools/include-what-you-use -p1 < $TP_DIR/patches/llvm-iwyu-llvm-6-compat.patch" \
+ "patch -d projects/compiler-rt -p1 < $TP_DIR/patches/llvm-tsan-disable-trace-switching-after-multithreaded-for.patch"
LZ4_PATCHLEVEL=1
fetch_and_patch \
http://git-wip-us.apache.org/repos/asf/kudu/blob/60acdb99/thirdparty/patches/llvm-tsan-disable-trace-switching-after-multithreaded-for.patch
----------------------------------------------------------------------
diff --git a/thirdparty/patches/llvm-tsan-disable-trace-switching-after-multithreaded-for.patch b/thirdparty/patches/llvm-tsan-disable-trace-switching-after-multithreaded-for.patch
new file mode 100644
index 0000000..4e4cda7
--- /dev/null
+++ b/thirdparty/patches/llvm-tsan-disable-trace-switching-after-multithreaded-for.patch
@@ -0,0 +1,53 @@
+From e45c6d51547e764174b50492b39b66bea2adbc90 Mon Sep 17 00:00:00 2001
+From: Dmitry Vyukov <dv...@google.com>
+Date: Mon, 30 Apr 2018 07:28:45 +0000
+Subject: [PATCH 1/1] tsan: disable trace switching after multithreaded fork
+
+The problem is reported in:
+https://github.com/google/sanitizers/issues/945
+
+We already disable as much as possible after multithreaded fork,
+trace switching is last place that can hang due to basic
+operations (memory accesses, function calls).
+Disable it too.
+
+
+
+git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@331163 91177308-0d34-0410-b5e6-96231b3b80d8
+---
+ lib/tsan/rtl/tsan_rtl.cc | 4 ++++
+ lib/tsan/rtl/tsan_rtl.h | 2 ++
+ 2 files changed, 6 insertions(+)
+
+diff --git a/lib/tsan/rtl/tsan_rtl.cc b/lib/tsan/rtl/tsan_rtl.cc
+index 17b8209..404fb80 100644
+--- a/lib/tsan/rtl/tsan_rtl.cc
++++ b/lib/tsan/rtl/tsan_rtl.cc
+@@ -547,6 +547,10 @@ u32 CurrentStackId(ThreadState *thr, uptr pc) {
+ }
+
+ void TraceSwitch(ThreadState *thr) {
++#if !SANITIZER_GO
++ if (ctx->after_multithreaded_fork)
++ return;
++#endif
+ thr->nomalloc++;
+ Trace *thr_trace = ThreadTrace(thr->tid);
+ Lock l(&thr_trace->mtx);
+diff --git a/lib/tsan/rtl/tsan_rtl.h b/lib/tsan/rtl/tsan_rtl.h
+index 7d44cca..5e2a745 100644
+--- a/lib/tsan/rtl/tsan_rtl.h
++++ b/lib/tsan/rtl/tsan_rtl.h
+@@ -520,7 +520,9 @@ struct Context {
+ Context();
+
+ bool initialized;
++#if !SANITIZER_GO
+ bool after_multithreaded_fork;
++#endif
+
+ MetaMap metamap;
+
+--
+2.7.4
+
[3/6] kudu git commit: KUDU-2399: Support IS NULL / IS NOT NULL
predicates in Python
Posted by gr...@apache.org.
KUDU-2399: Support IS NULL / IS NOT NULL predicates in Python
Add is_null and is_not_null predicates in kudu-python. Had to change
the KuduTestBase and TestScanner to default to None and not replace
None with 'nothing' so I could write tests for null predicates.
I also fixed the test_scan_rows_string_predicate_and_projection test
since it was missing the projections and failing.
Change-Id: I52a22d6d8a8fe9a6049270596eb131799c37f82f
Reviewed-on: http://gerrit.cloudera.org:8080/10072
Tested-by: Kudu Jenkins
Reviewed-by: David Ribeiro Alves <da...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/3bf125f4
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/3bf125f4
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/3bf125f4
Branch: refs/heads/master
Commit: 3bf125f469a2929f7fe20e4e31f073376f37575d
Parents: 9173015
Author: Andy Stadtler <an...@phdata.io>
Authored: Sat Apr 14 14:06:29 2018 -0500
Committer: Grant Henke <gr...@apache.org>
Committed: Mon Apr 30 19:58:11 2018 +0000
----------------------------------------------------------------------
python/kudu/client.pyx | 58 ++++++++++++++++++++++++++++++++++
python/kudu/libkudu_client.pxd | 2 ++
python/kudu/tests/common.py | 2 +-
python/kudu/tests/test_scanner.py | 33 +++++++++++++++++--
python/kudu/tests/util.py | 8 +----
5 files changed, 93 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/3bf125f4/python/kudu/client.pyx
----------------------------------------------------------------------
diff --git a/python/kudu/client.pyx b/python/kudu/client.pyx
index 799b44e..5997bee 100644
--- a/python/kudu/client.pyx
+++ b/python/kudu/client.pyx
@@ -995,6 +995,64 @@ cdef class Column:
return result
+ def is_not_null(Column self):
+ """
+ Creates a new IsNotNullPredicate for the Column which can be used for scanners
+ on this table.
+
+ Examples
+ --------
+ scanner.add_predicate(table[col_name].is_not_null())
+
+ Returns
+ -------
+ pred : Predicate
+ """
+
+ cdef:
+ KuduPredicate* pred
+ Slice col_name_slice
+ Predicate result
+ object _name = tobytes(self.name)
+
+ col_name_slice = Slice(<char*> _name, len(_name))
+
+ pred = (self.parent.ptr()
+ .NewIsNotNullPredicate(col_name_slice))
+
+ result = Predicate()
+ result.init(pred)
+
+ return result
+
+ def is_null(Column self):
+ """
+ Creates a new IsNullPredicate for the Column which can be used for scanners on this table.
+
+ Examples
+ --------
+ scanner.add_predicate(table[col_name].is_null())
+
+ Returns
+ -------
+ pred : Predicate
+ """
+
+ cdef:
+ KuduPredicate* pred
+ Slice col_name_slice
+ Predicate result
+ object _name = tobytes(self.name)
+
+ col_name_slice = Slice(<char*> _name, len(_name))
+
+ pred = (self.parent.ptr()
+ .NewIsNullPredicate(col_name_slice))
+
+ result = Predicate()
+ result.init(pred)
+
+ return result
class Partitioning(object):
""" Argument to Client.create_table(...) to describe table partitioning. """
http://git-wip-us.apache.org/repos/asf/kudu/blob/3bf125f4/python/kudu/libkudu_client.pxd
----------------------------------------------------------------------
diff --git a/python/kudu/libkudu_client.pxd b/python/kudu/libkudu_client.pxd
index a489784..5cd7731 100644
--- a/python/kudu/libkudu_client.pxd
+++ b/python/kudu/libkudu_client.pxd
@@ -609,6 +609,8 @@ cdef extern from "kudu/client/client.h" namespace "kudu::client" nogil:
C_KuduValue* value);
KuduPredicate* NewInListPredicate(const Slice& col_name,
vector[C_KuduValue*]* values)
+ KuduPredicate* NewIsNotNullPredicate(const Slice& col_name)
+ KuduPredicate* NewIsNullPredicate(const Slice& col_name)
KuduClient* client()
# const PartitionSchema& partition_schema()
http://git-wip-us.apache.org/repos/asf/kudu/blob/3bf125f4/python/kudu/tests/common.py
----------------------------------------------------------------------
diff --git a/python/kudu/tests/common.py b/python/kudu/tests/common.py
index 9596eaa..e1629ea 100644
--- a/python/kudu/tests/common.py
+++ b/python/kudu/tests/common.py
@@ -114,7 +114,7 @@ class KuduTestBase(object):
builder = kudu.schema_builder()
builder.add_column('key', kudu.int32, nullable=False)
builder.add_column('int_val', kudu.int32)
- builder.add_column('string_val', kudu.string, default='nothing')
+ builder.add_column('string_val', kudu.string, default=None)
builder.add_column('unixtime_micros_val', kudu.unixtime_micros)
builder.set_primary_keys(['key'])
http://git-wip-us.apache.org/repos/asf/kudu/blob/3bf125f4/python/kudu/tests/test_scanner.py
----------------------------------------------------------------------
diff --git a/python/kudu/tests/test_scanner.py b/python/kudu/tests/test_scanner.py
index 00fe7a8..a4d0091 100644
--- a/python/kudu/tests/test_scanner.py
+++ b/python/kudu/tests/test_scanner.py
@@ -70,10 +70,9 @@ class TestScanner(TestScanBase):
def test_scan_rows_string_predicate_and_projection(self):
scanner = self.table.scanner()
- scanner.set_projected_column_names(['key', 'string_val'])
-
sv = self.table['string_val']
+ scanner.set_projected_column_indexes([0, 2])
scanner.add_predicates([sv >= 'hello_20',
sv <= 'hello_22'])
@@ -102,6 +101,36 @@ class TestScanner(TestScanBase):
self.assertEqual(tuples, [self.tuples[98]])
+ def test_scan_rows_is_not_null_predicate(self):
+ """
+ Test scanner with an IsNotNull predicate on string_val column
+ """
+ pred = self.table['string_val'].is_not_null()
+ scanner = self.table.scanner()
+ scanner.add_predicate(pred)
+ scanner.open()
+
+ tuples = scanner.read_all_tuples()
+
+ rows = [i for i in range(100) if i % 2 == 0]
+
+ self.assertEqual(sorted(tuples), [self.tuples[i] for i in rows])
+
+ def test_scan_rows_is_null_predicate(self):
+ """
+ Test scanner with an IsNull predicate on string_val column
+ """
+ pred = self.table['string_val'].is_null()
+ scanner = self.table.scanner()
+ scanner.add_predicate(pred)
+ scanner.open()
+
+ tuples = scanner.read_all_tuples()
+
+ rows = [i for i in range(100) if i % 2 != 0]
+
+ self.assertEqual(sorted(tuples), [self.tuples[i] for i in rows])
+
def test_index_projection_with_schema(self):
scanner = self.table.scanner()
scanner.set_projected_column_indexes([0, 1])
http://git-wip-us.apache.org/repos/asf/kudu/blob/3bf125f4/python/kudu/tests/util.py
----------------------------------------------------------------------
diff --git a/python/kudu/tests/util.py b/python/kudu/tests/util.py
index a6779fc..6823b3a 100644
--- a/python/kudu/tests/util.py
+++ b/python/kudu/tests/util.py
@@ -56,13 +56,7 @@ class TestScanBase(KuduTestBase, unittest.TestCase):
session.flush()
self.table = table
- self.tuples = []
-
- # Replace missing values w/ defaults to test default values.
- for tuple in tuples:
- if tuple[2] == None:
- tuple = (tuple[0], tuple[1], 'nothing', tuple[3])
- self.tuples.append(tuple)
+ self.tuples = tuples
# Create table to test all types
# for various predicate tests
[4/6] kudu git commit: java: prohibit use of a KuduTable from an
unassociated KuduClient
Posted by gr...@apache.org.
java: prohibit use of a KuduTable from an unassociated KuduClient
This fixes a request-tracking issue with the following code
anti-pattern in which a KuduTable associated with one client is used to
create operations applied to another client's session:
KuduClient client1 = KuduClientBuilder....newClient();
KuduTable t = client1.openTable(...);
KuduClient client2 = KuduClientBuilder....newClient();
KuduSession s = client2.newSession();
s.apply(t.newUpdate());
This would cause sequence numbers to be generated out of the session's
client's RequestTracker, but then marked complete in the operation's
client's RequestTracker. This could cause any number of issues:
- a cache "hit" on the server side might cause an operation to get an
unrelated operation's response
- a cache "miss" on the server side might result in an operation
incorrectly being marked as already-responded and garbage collected,
causing it to fail.
This patch adds a Preconditions check for this issue and fixes some tests
where it was triggered.
Change-Id: Ib4a977f36b9c7ba3758322a2216ce90208b5d014
Reviewed-on: http://gerrit.cloudera.org:8080/7362
Reviewed-by: David Ribeiro Alves <da...@gmail.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/3adac316
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/3adac316
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/3adac316
Branch: refs/heads/master
Commit: 3adac3160039b5cc5d60cba17ee134aa6c1c23cc
Parents: 3bf125f
Author: Todd Lipcon <to...@apache.org>
Authored: Wed Jul 5 19:11:02 2017 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Apr 30 20:55:35 2018 +0000
----------------------------------------------------------------------
.../apache/kudu/client/AsyncKuduSession.java | 3 +
.../java/org/apache/kudu/client/KuduRpc.java | 4 +-
.../java/org/apache/kudu/client/Operation.java | 3 +
.../org/apache/kudu/client/RequestTracker.java | 8 ++-
.../org/apache/kudu/client/ITClientStress.java | 60 ++++++++----------
.../org/apache/kudu/client/TestKuduClient.java | 3 +-
.../apache/kudu/client/TestRequestTracker.java | 5 +-
.../kudu/client/TestScannerMultiTablet.java | 65 ++++++++++++++++----
.../org/apache/kudu/client/TestTimeouts.java | 3 +-
9 files changed, 100 insertions(+), 54 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
index 01d6fc8..94ca290 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java
@@ -529,6 +529,9 @@ public class AsyncKuduSession implements SessionConfiguration {
*/
public Deferred<OperationResponse> apply(final Operation operation) throws KuduException {
Preconditions.checkNotNull(operation, "Can not apply a null operation");
+ Preconditions.checkArgument(operation.getTable().getAsyncClient() == client,
+ "Applied operations must be created from a KuduTable instance opened " +
+ "from the same client that opened this KuduSession");
// Freeze the row so that the client can not concurrently modify it while it is in flight.
operation.getRow().freeze();
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java
index dbecaa2..0083140 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java
@@ -235,7 +235,9 @@ public abstract class KuduRpc<R> {
}
deferred = null;
attempt = 0;
- if (isRequestTracked()) {
+ // If the subclass is a "tracked RPC" unregister it, unless it never
+ // got to the point of being registered.
+ if (isRequestTracked() && sequenceId != RequestTracker.NO_SEQ_NO) {
table.getAsyncClient().getRequestTracker().rpcCompleted(sequenceId);
sequenceId = RequestTracker.NO_SEQ_NO;
}
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java b/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
index 3a9a294..3d04450 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
@@ -42,6 +42,9 @@ import org.apache.kudu.util.Pair;
/**
* Base class for the RPCs that related to WriteRequestPB. It contains almost all the logic
* and knows how to serialize its child classes.
+ *
+ * TODO(todd): this should not extend KuduRpc. Rather, we should make single-operation writes
+ * just use a Batch instance with a single operation in it.
*/
@InterfaceAudience.Public
@InterfaceStability.Evolving
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java b/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
index 2a8bc2c..7db88fa 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
@@ -73,12 +73,16 @@ public class RequestTracker {
}
/**
- * Marks the given sequence id as complete. This operation is idempotent.
+ * Marks the given sequence id as complete. The provided sequence ID must be a valid
+ * number that was previously returned by {@link #newSeqNo()}. It is illegal to call
+ * this method twice with the same sequence number.
* @param sequenceId the sequence id to mark as complete
*/
public void rpcCompleted(long sequenceId) {
+ assert sequenceId != NO_SEQ_NO;
synchronized (lock) {
- incompleteRpcs.remove(sequenceId);
+ boolean removed = incompleteRpcs.remove(sequenceId);
+ assert(removed) : "Could not remove seqid " + sequenceId + " from request tracker";
}
}
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java b/java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java
index 0922fe7..1c78e55 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java
@@ -129,42 +129,36 @@ public class ITClientStress extends BaseKuduTest {
final KuduTable table = createTable(TABLE_NAME, basicSchema,
getBasicCreateTableOptions());
final AtomicInteger numUpserted = new AtomicInteger(0);
- try (final KuduClient client =
- new KuduClient.KuduClientBuilder(masterAddresses)
- .defaultAdminOperationTimeoutMs(DEFAULT_SLEEP)
- .build()) {
-
- runTasks(NUM_THREADS, SECONDS_TO_RUN, new Supplier<Callable<Void>>() {
- @Override
- public Callable<Void> get() {
- return new Callable<Void>() {
- @Override
- public Void call() throws Exception {
- KuduSession s = client.newSession();
- s.setFlushMode(FlushMode.AUTO_FLUSH_SYNC);
- try {
- for (int i = 0; i < 100; i++) {
- Upsert u = table.newUpsert();
- u.getRow().addInt(0, i);
- u.getRow().addInt(1, 12345);
- u.getRow().addInt(2, 3);
- u.getRow().setNull(3);
- u.getRow().addBoolean(4, false);
- OperationResponse apply = s.apply(u);
- if (apply.hasRowError()) {
- throw new AssertionError(apply.getRowError().toString());
- }
- numUpserted.incrementAndGet();
+ runTasks(NUM_THREADS, SECONDS_TO_RUN, new Supplier<Callable<Void>>() {
+ @Override
+ public Callable<Void> get() {
+ return new Callable<Void>() {
+ @Override
+ public Void call() throws Exception {
+ KuduSession s = syncClient.newSession();
+ s.setFlushMode(FlushMode.AUTO_FLUSH_SYNC);
+ try {
+ for (int i = 0; i < 100; i++) {
+ Upsert u = table.newUpsert();
+ u.getRow().addInt(0, i);
+ u.getRow().addInt(1, 12345);
+ u.getRow().addInt(2, 3);
+ u.getRow().setNull(3);
+ u.getRow().addBoolean(4, false);
+ OperationResponse apply = s.apply(u);
+ if (apply.hasRowError()) {
+ throw new AssertionError(apply.getRowError().toString());
}
- } finally {
- s.close();
+ numUpserted.incrementAndGet();
}
- return null;
+ } finally {
+ s.close();
}
- };
- }
- });
- }
+ return null;
+ }
+ };
+ }
+ });
LOG.info("Upserted {} rows", numUpserted.get());
}
}
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
index 741237d..6ff02ad 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
@@ -524,7 +524,8 @@ public class TestKuduClient extends BaseKuduTest {
while (scanner.hasMoreRows()) {
count += scanner.nextRows().getNumRows();
}
- assertEquals(Math.min(num_rows, limit), count);
+ assertEquals(String.format("Limit %d returned %d/%d rows", limit, count, num_rows),
+ Math.min(num_rows, limit), count);
}
// Now test with limits for async scanners.
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
index c75c4fb..980f447 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
@@ -57,9 +57,8 @@ public class TestRequestTracker {
tracker.rpcCompleted(5);
assertEquals(2, tracker.firstIncomplete());
- // Mark the first half as complete.
- // Note that we're also testing that rpcCompleted is idempotent.
- for (int i = 1; i < max / 2; i++) {
+ // Mark 2-4 inclusive as complete.
+ for (int i = 2; i <= 4; i++) {
tracker.rpcCompleted(i);
}
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/test/java/org/apache/kudu/client/TestScannerMultiTablet.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestScannerMultiTablet.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestScannerMultiTablet.java
index 22a5bed..7a7fb7e 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestScannerMultiTablet.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestScannerMultiTablet.java
@@ -27,7 +27,10 @@ import java.util.ArrayList;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import com.stumbleupon.async.Deferred;
+
+import org.junit.Before;
import org.junit.BeforeClass;
+import org.junit.Ignore;
import org.junit.Test;
import org.apache.kudu.client.Client.ScanTokenPB;
@@ -41,7 +44,13 @@ public class TestScannerMultiTablet extends BaseKuduTest {
TestScannerMultiTablet.class.getName()+"-"+System.currentTimeMillis();
private static Schema schema = getSchema();
- private static KuduTable table;
+
+ /**
+ * The timestamp after inserting the rows into the test table during
+ * setupBeforeClass().
+ */
+ private static long beforeClassWriteTimestamp;
+ private KuduTable table;
@BeforeClass
public static void setUpBeforeClass() throws Exception {
@@ -59,8 +68,7 @@ public class TestScannerMultiTablet extends BaseKuduTest {
createTable(TABLE_NAME, schema, builder);
- table = openTable(TABLE_NAME);
-
+ KuduTable table = openTable(TABLE_NAME);
AsyncKuduSession session = client.newSession();
session.setFlushMode(AsyncKuduSession.FlushMode.AUTO_FLUSH_SYNC);
@@ -81,6 +89,18 @@ public class TestScannerMultiTablet extends BaseKuduTest {
d.join(DEFAULT_SLEEP);
}
}
+
+ beforeClassWriteTimestamp = client.getLastPropagatedTimestamp();
+ }
+
+ @Before
+ public void setup() throws Exception {
+ // Reset the clients in order to clear the propagated timestamp, which may
+ // have been set if other test cases ran before this one. This ensures
+ // that all tests set their own state.
+ resetClients();
+ // Reopen the table using the reset client.
+ table = openTable(TABLE_NAME);
}
// Test various combinations of start/end row keys.
@@ -240,21 +260,47 @@ public class TestScannerMultiTablet extends BaseKuduTest {
assertEquals(9, rowCount);
}
+ // Regression test for KUDU-2415.
+ // Scanning a never-written-to tablet from a fresh client with no propagated
+ // timestamp in "read-your-writes' mode should not fail.
+ @Test(timeout = 100000)
+ @Ignore("TODO(KUDU-2415)") // not fixed yet!
+ public void testReadYourWritesFreshClientFreshTable() throws Exception {
+ // NOTE: this test fails because the first tablet in the table
+ // is empty and has never been written to.
+
+ // Perform scan in READ_YOUR_WRITES mode. Before the scan, verify that the
+ // propagated timestamp is unset, since this is a fresh client.
+ AsyncKuduScanner scanner = client.newScannerBuilder(table)
+ .readMode(AsyncKuduScanner.ReadMode.READ_YOUR_WRITES)
+ .build();
+ KuduScanner syncScanner = new KuduScanner(scanner);
+ assertEquals(scanner.getReadMode(), syncScanner.getReadMode());
+ assertEquals(AsyncKuduClient.NO_TIMESTAMP, client.getLastPropagatedTimestamp());
+ assertEquals(AsyncKuduClient.NO_TIMESTAMP, scanner.getSnapshotTimestamp());
+
+ assertEquals(9, countRowsInScan(syncScanner));
+ }
+
// Test multi tablets scan in READ_YOUR_WRITES mode for both AUTO_FLUSH_SYNC
// (single operation) and MANUAL_FLUSH (batches) flush modes to ensure
// client-local read-your-writes.
@Test(timeout = 100000)
public void testReadYourWrites() throws Exception {
+ long preTs = beforeClassWriteTimestamp;
+
+ // Update the propagated timestamp to ensure we see the rows written
+ // in the constructor.
+ syncClient.updateLastPropagatedTimestamp(preTs);
+
// Perform scan in READ_YOUR_WRITES mode. Before the scan, verify that the
- // propagated timestamp is set via previous write session while snapshot
- // timestamp is not.
+ // scanner timestamp is not yet set. It will get set only once the scan
+ // is opened.
AsyncKuduScanner scanner = client.newScannerBuilder(table)
.readMode(AsyncKuduScanner.ReadMode.READ_YOUR_WRITES)
.build();
KuduScanner syncScanner = new KuduScanner(scanner);
assertEquals(scanner.getReadMode(), syncScanner.getReadMode());
- long preTs = client.getLastPropagatedTimestamp();
- assertNotEquals(AsyncKuduClient.NO_TIMESTAMP, client.getLastPropagatedTimestamp());
assertEquals(AsyncKuduClient.NO_TIMESTAMP, scanner.getSnapshotTimestamp());
assertEquals(9, countRowsInScan(syncScanner));
@@ -298,9 +344,6 @@ public class TestScannerMultiTablet extends BaseKuduTest {
@Test(timeout = 100000)
public void testScanPropagatesLatestTimestamp() throws Exception {
- // Reset the clients in order to clear the propagated timestamp, which may
- // have been set if other test cases ran before this one.
- resetClients();
AsyncKuduScanner scanner = client.newScannerBuilder(table).build();
// Initially, the client does not have the timestamp set.
@@ -339,8 +382,6 @@ public class TestScannerMultiTablet extends BaseKuduTest {
@Test(timeout = 100000)
public void testScanTokenPropagatesTimestamp() throws Exception {
- resetClients();
-
// Initially, the client does not have the timestamp set.
assertEquals(AsyncKuduClient.NO_TIMESTAMP, client.getLastPropagatedTimestamp());
assertEquals(KuduClient.NO_TIMESTAMP, syncClient.getLastPropagatedTimestamp());
http://git-wip-us.apache.org/repos/asf/kudu/blob/3adac316/java/kudu-client/src/test/java/org/apache/kudu/client/TestTimeouts.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestTimeouts.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestTimeouts.java
index 5eca5d2..462f49e 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestTimeouts.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestTimeouts.java
@@ -46,11 +46,10 @@ public class TestTimeouts extends BaseKuduTest {
}
createTable(TABLE_NAME, basicSchema, getBasicCreateTableOptions());
- KuduTable table = openTable(TABLE_NAME);
+ KuduTable table = lowTimeoutsClient.openTable(TABLE_NAME);
KuduSession lowTimeoutSession = lowTimeoutsClient.newSession();
-
OperationResponse response = lowTimeoutSession.apply(createBasicSchemaInsert(table, 1));
assertTrue(response.hasRowError());
assertTrue(response.getRowError().getErrorStatus().isTimedOut());
[5/6] kudu git commit: java: fix error-prone 'StringSplit' pattern[1]
Posted by gr...@apache.org.
java: fix error-prone 'StringSplit' pattern[1]
This should be a non-functional change.
[1] https://github.com/google/error-prone/blob/master/docs/bugpattern/StringSplit.md
Change-Id: I5379ca32be843374fd4a9f7213d096998e436baa
Reviewed-on: http://gerrit.cloudera.org:8080/10196
Reviewed-by: Grant Henke <gr...@apache.org>
Tested-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/eac9d172
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/eac9d172
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/eac9d172
Branch: refs/heads/master
Commit: eac9d172de571e72d0a2652f891dfe9e43c373aa
Parents: 3adac31
Author: Todd Lipcon <to...@apache.org>
Authored: Wed Apr 25 11:40:47 2018 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Apr 30 20:56:02 2018 +0000
----------------------------------------------------------------------
.../apache/kudu/mapreduce/tools/IntegrationTestBigLinkedList.java | 2 +-
.../src/test/java/org/apache/kudu/mapreduce/tools/ITExportCsv.java | 2 +-
.../src/main/java/org/apache/kudu/util/SecurityUtil.java | 2 +-
.../src/test/java/org/apache/kudu/client/TestConnectToCluster.java | 2 +-
.../src/test/java/org/apache/kudu/client/TestKuduClient.java | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/eac9d172/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/IntegrationTestBigLinkedList.java
----------------------------------------------------------------------
diff --git a/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/IntegrationTestBigLinkedList.java b/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/IntegrationTestBigLinkedList.java
index 0de052b..e3a1a48 100644
--- a/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/IntegrationTestBigLinkedList.java
+++ b/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/IntegrationTestBigLinkedList.java
@@ -1373,7 +1373,7 @@ public class IntegrationTestBigLinkedList extends Configured implements Tool {
}
System.out.println("Running Walker with args:" + Arrays.deepToString(args));
- String[] keys = args[0].split(",");
+ String[] keys = args[0].split(",", -1);
if (keys.length != 2) {
System.err.println("The row key must be formatted like key1,key2");
return 1;
http://git-wip-us.apache.org/repos/asf/kudu/blob/eac9d172/java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITExportCsv.java
----------------------------------------------------------------------
diff --git a/java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITExportCsv.java b/java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITExportCsv.java
index 984ec63..93a5e13 100644
--- a/java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITExportCsv.java
+++ b/java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITExportCsv.java
@@ -78,7 +78,7 @@ public class ITExportCsv extends BaseKuduTest {
String csvContent = readCsvFile(new File(testHome + "/exportdata/part-m-00001"));
assertEquals(csvContent.split("\n").length,3);
- assertEquals(csvContent.split("\n")[0].split("\t")[3],"a string");
+ assertEquals(csvContent.split("\n", -1)[0].split("\t", -1)[3],"a string");
}
private String readCsvFile(File data) throws IOException {
http://git-wip-us.apache.org/repos/asf/kudu/blob/eac9d172/java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java b/java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java
index 34ea70c..4ec5734 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java
@@ -143,7 +143,7 @@ public abstract class SecurityUtil {
// The signature algorithm name is a string like 'SHA256withRSA'.
// There's no API available to actually find just the digest algorithm,
// so we resort to some hackery.
- String[] components = sigAlg.split("with");
+ String[] components = sigAlg.split("with", 2);
String digestAlg = CERT_DIGEST_TO_MESSAGE_DIGEST.get(components[0].toUpperCase());
if (digestAlg == null) {
// RFC 5929: if the certificate's signatureAlgorithm uses no hash functions or
http://git-wip-us.apache.org/repos/asf/kudu/blob/eac9d172/java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectToCluster.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectToCluster.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectToCluster.java
index bb94bce..6bda971 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectToCluster.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectToCluster.java
@@ -83,7 +83,7 @@ public class TestConnectToCluster {
.build();
int successes = 0;
try {
- String[] masterAddrs = cluster.getMasterAddresses().split(",");
+ String[] masterAddrs = cluster.getMasterAddresses().split(",", -1);
assertEquals(3, masterAddrs.length);
for (String masterAddr : masterAddrs) {
KuduClient c = null;
http://git-wip-us.apache.org/repos/asf/kudu/blob/eac9d172/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
index 6ff02ad..56a4da7 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
@@ -247,7 +247,7 @@ public class TestKuduClient extends BaseKuduTest {
"STRING key=r3, STRING c1=NULL, STRING c2=def, STRING c3=c, STRING c4=def");
for (String row : rows) {
try {
- String[] fields = row.split(",");
+ String[] fields = row.split(",", -1);
Insert insert = table.newInsert();
for (int i = 0; i < fields.length; i++) {
if (fields[i].equals("-")) { // leave unset
[6/6] kudu git commit: java: fix minor synchronization issues exposed
by error-prone
Posted by gr...@apache.org.
java: fix minor synchronization issues exposed by error-prone
* Unsynchronized access to 'sessions' in AsyncKuduClient.closeAllSessions().
This isn't likely to cause problems since it only happens on 'close'
and would only race if a new session was concurrently started.
* Add missing GuardedBy annotations in a few spots (non-functional)
* Fix synchronization in TableLocationsCache.toString() to properly
acquire the lock. Unlikely to cause problems in practice since we only
call toString() on this in some rare trace messages (if that).
* Fix synchronization in FakeDNS (test-only code)
Change-Id: I1c737f59928f393883d198872419e8832dfff006
Reviewed-on: http://gerrit.cloudera.org:8080/10199
Reviewed-by: Grant Henke <gr...@apache.org>
Tested-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/69fb524c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/69fb524c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/69fb524c
Branch: refs/heads/master
Commit: 69fb524c413ee39f87e8a908171274ba9727e194
Parents: eac9d17
Author: Todd Lipcon <to...@apache.org>
Authored: Wed Apr 25 11:47:45 2018 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Apr 30 20:56:51 2018 +0000
----------------------------------------------------------------------
.../java/org/apache/kudu/client/AsyncKuduClient.java | 2 +-
.../src/main/java/org/apache/kudu/client/Connection.java | 1 +
.../java/org/apache/kudu/client/TableLocationsCache.java | 7 ++++++-
.../src/test/java/org/apache/kudu/client/FakeDNS.java | 11 ++++++++---
4 files changed, 16 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/69fb524c/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
index 383df92..83d0684 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
@@ -2090,7 +2090,7 @@ public class AsyncKuduClient implements AutoCloseable {
synchronized (sessions) {
copyOfSessions = new HashSet<>(sessions);
}
- if (sessions.isEmpty()) {
+ if (copyOfSessions.isEmpty()) {
return Deferred.fromResult(null);
}
// Guaranteed that we'll have at least one session to close.
http://git-wip-us.apache.org/repos/asf/kudu/blob/69fb524c/java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java b/java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java
index 5401000..5d3155b 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java
@@ -722,6 +722,7 @@ class Connection extends SimpleChannelUpstreamHandler {
}
/** Initiate opening TCP connection to the server. */
+ @GuardedBy("lock")
private void connect() {
Preconditions.checkState(lock.isHeldByCurrentThread());
Preconditions.checkState(state == State.NEW);
http://git-wip-us.apache.org/repos/asf/kudu/blob/69fb524c/java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java b/java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java
index d8e5941..03d80dc 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java
@@ -234,7 +234,12 @@ class TableLocationsCache {
@Override
public String toString() {
- return entries.values().toString();
+ rwl.readLock().lock();
+ try {
+ return entries.values().toString();
+ } finally {
+ rwl.readLock().unlock();
+ }
}
/**
http://git-wip-us.apache.org/repos/asf/kudu/blob/69fb524c/java/kudu-client/src/test/java/org/apache/kudu/client/FakeDNS.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/test/java/org/apache/kudu/client/FakeDNS.java b/java/kudu-client/src/test/java/org/apache/kudu/client/FakeDNS.java
index 25ff5ed..db9cd04 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/FakeDNS.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/FakeDNS.java
@@ -91,8 +91,10 @@ public class FakeDNS {
@Override
public InetAddress[] lookupAllHostAddr(String host)
throws UnknownHostException {
-
- InetAddress inetAddress = forwardResolutions.get(host);
+ InetAddress inetAddress;
+ synchronized(FakeDNS.this) {
+ inetAddress = forwardResolutions.get(host);
+ }
if (inetAddress != null) {
return new InetAddress[]{inetAddress};
}
@@ -107,7 +109,10 @@ public class FakeDNS {
return InetAddresses.toAddrString(InetAddress.getByAddress(addr));
}
- String hostname = reverseResolutions.get(InetAddress.getByAddress(addr));
+ String hostname;
+ synchronized (FakeDNS.this) {
+ hostname = reverseResolutions.get(InetAddress.getByAddress(addr));
+ }
if (hostname != null) {
return hostname;
}
[2/6] kudu git commit: benchmarks: update parameters for mt-bloomfile
benchmark
Posted by gr...@apache.org.
benchmarks: update parameters for mt-bloomfile benchmark
The old parameters were somewhat unrealistic:
- The default bloom block size is 4kb, but it was using 32kb.
- The cache capacity was set to 1MB, which meant that each shard on a
multicore system was only 50KB or less. With the new optimization that
allows 1% slack in MemTracker accounting, that slack was only
500bytes or less. With 4kb bloom blocks, that optimization was
effectively disabled.
The new parameters are slightly more realistic while keeping the
performance reasonable: 100MB cache capacity (>1MB per shard even on 88
core system) and 4kb blocks. Runtime is still reasonable - 4-5 seconds
for the query phase. The number of unique keys is bumped up to 150M to
ensure cache churn.
Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Reviewed-on: http://gerrit.cloudera.org:8080/10242
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/91730158
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/91730158
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/91730158
Branch: refs/heads/master
Commit: 9173015808074c6f771d08be25812446f6a87d48
Parents: 60acdb9
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Apr 30 10:21:04 2018 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Apr 30 18:58:22 2018 +0000
----------------------------------------------------------------------
src/kudu/scripts/benchmarks.sh | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/91730158/src/kudu/scripts/benchmarks.sh
----------------------------------------------------------------------
diff --git a/src/kudu/scripts/benchmarks.sh b/src/kudu/scripts/benchmarks.sh
index 1c9f18c..346debe 100755
--- a/src/kudu/scripts/benchmarks.sh
+++ b/src/kudu/scripts/benchmarks.sh
@@ -208,10 +208,12 @@ run_benchmarks() {
done
# run mt-bloomfile-test 5 times. 20-30 seconds per run.
- # The block cache is set to 1MB to generate churn.
+ # We use 150M keys so the total bloom space is >150MB, and then
+ # set the cache capacity to be only 100M to ensure that the cache
+ # churns a lot.
for i in $(seq 1 $NUM_SAMPLES); do
- ./build/latest/bin/mt-bloomfile-test --benchmark_queries=2000000 --bloom_size_bytes=32768 \
- --n_keys=5000000 --block_cache_capacity_mb=1 &> $LOGDIR/$MT_BLOOM_TEST$i.log
+ ./build/latest/bin/mt-bloomfile-test --benchmark_queries=2000000 --bloom_size_bytes=4096 \
+ --n_keys=150000000 --block_cache_capacity_mb=100 &> $LOGDIR/$MT_BLOOM_TEST$i.log
done
# run wire_protocol-test 5 times. 6 seconds per run