You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2023/07/19 22:25:32 UTC
[GitHub] [accumulo] keith-turner opened a new pull request, #3639: Adds IT that verifies scans see data written by concurrent writers
keith-turner opened a new pull request, #3639:
URL: https://github.com/apache/accumulo/pull/3639
The IT ensures that when there are concurrent writes, scans, and table operations that scans always see any data written before the scan started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1269926881
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
Review Comment:
Yeah I think there is value in running it longer, but maybe not in the IT. We definitely want to know if does not work on a short run in the IT. I think I will add a main method that allows running it. I think it would be good to have as an IT and something that is easy to run for longer periods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1281571570
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,697 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.BatchScanner;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.ScannerBase;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.client.rfile.RFile;
+import org.apache.accumulo.core.client.rfile.RFileWriter;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorEnvironment;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Collections2;
+import com.google.common.collect.MoreCollectors;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
Review Comment:
I don't think it makes sense to add this to the sunny tests, as it's not testing base functionality, but it's testing more complicated edge cases that can be left in the larger test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner merged pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #3639:
URL: https://github.com/apache/accumulo/pull/3639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#issuecomment-1645985243
I added merge, bulk import, compaction filtering, and batch scan to the test in 7cbc635. I also added some debugging tips based on a problem I ran into where bulk importing deletes was causing the test to fail. Spent hours trying to track down the cause and finally developed a technique that made tracking the problem down really quick. I documented the technique [in the code](https://github.com/apache/accumulo/blob/7cbc6356fecb569a713094dba779f5e54c9301c9/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java#L93-L108) and left a [utility method](https://github.com/apache/accumulo/blob/7cbc6356fecb569a713094dba779f5e54c9301c9/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java#L668-L696) related to the technique in the test.
The problem I ran into is documented [here in the test](https://github.com/apache/accumulo/blob/7cbc6356fecb569a713094dba779f5e54c9301c9/test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java#L488-L496). The problem is caused by the fact that major compactions no longer force a flush, this is a change made some time in the past, trying to find an issue for that. Not flushing before a full major compaction is ok when only doing live ingest or only doing bulk import, but for a mix of bulk import and live ingest not flushing before full major compaction has a consequence as I discovered in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "EdColeman (via GitHub)" <gi...@apache.org>.
EdColeman commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1270950955
##########
core/src/main/java/org/apache/accumulo/core/metadata/schema/DataFileValue.java:
##########
@@ -105,7 +105,7 @@ public int hashCode() {
@Override
public String toString() {
- return size + " " + numEntries;
+ return size + " " + numEntries + " " + time;
Review Comment:
I realize that this adds to the existing pattern - but would it be worth it to use a more standardized toString format that includes the field names? Otherwise, you need to read the code to understand that '12 1000 1234` is 'size: 12, numEntries 1000, time 1234'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1269924266
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
+
+ // let the threads know to exit
+ testContext.keepRunning.set(false);
+
+ for (Future<WriteStats> writeTask : writeTasks) {
+ var stats = writeTask.get();
+ log.debug(String.format("Wrote:%,d Deleted:%,d", stats.written, stats.deleted));
+ assertTrue(stats.written > 0);
+ assertTrue(stats.deleted > 0);
+ }
+
+ for (Future<ScanStats> scanTask : scanTasks) {
+ var stats = scanTask.get();
+ log.debug(String.format("Scanned:%,d verified:%,d", stats.scanned, stats.verified));
+ assertTrue(stats.verified > 0);
+ // These scans were running concurrently with writes, so a scan will see more data than what
+ // was written before the scan started.
+ assertTrue(stats.scanned > stats.verified);
+ }
+
+ log.debug(tableOpsTask.get());
+
+ var stats1 = scanData(testContext, new Range(), false);
+ var stats2 = scanData(testContext, new Range(), true);
+ log.debug(
+ String.format("Final scan, scanned:%,d verified:%,d", stats1.scanned, stats1.verified));
+ assertTrue(stats1.verified > 0);
+ // Should see all expected data now that there are no concurrent writes happening
+ assertEquals(stats1.scanned, stats1.verified);
+ assertEquals(stats2.scanned, stats1.scanned);
+ assertEquals(stats2.verified, stats1.verified);
+ } finally {
+ executor.shutdownNow();
+ }
+
+ }
+
+ /**
+ * Tracks what data has been written and deleted in an Accumulo table.
+ */
+ private static class DataTracker {
+ // ConcurrentLinkedQueue was chosen because its safe to iterate over concurrently w/o locking
+ private final Queue<DataSet> dataSets = new ConcurrentLinkedQueue<>();
+
+ /**
+ * Reserves data for scan so that it will not be deleted and returns the data that is expected
+ * to exist in the table at this point.
+ */
+ public ExpectedScanData beginScan() {
+ List<DataSet> reservedData = new ArrayList<>();
+
+ for (var dataSet : dataSets) {
+ if (dataSet.reserveForScan()) {
+ reservedData.add(dataSet);
+ }
+ }
+
+ return new ExpectedScanData(reservedData);
+ }
+
+ /**
+ * add new data that scans should expect to see
+ */
+ public void addExpectedData(List<Mutation> data) {
+ dataSets.add(new DataSet(data));
+ }
+
+ /**
+ * @return data to delete from the table that is not reserved for scans
+ */
+ public Iterable<Mutation> getDeletes() {
+ DataSet dataSet = dataSets.poll();
+
+ if (dataSet == null) {
+ return List.of();
+ }
+
+ dataSet.reserveForDelete();
+
+ return () -> dataSet.data.stream().map(m -> {
+ Mutation delMutation = new Mutation(m.getRow());
+ m.getUpdates()
+ .forEach(cu -> delMutation.putDelete(cu.getColumnFamily(), cu.getColumnQualifier()));
+ return delMutation;
+ }).iterator();
+ }
+
+ public long estimatedRows() {
+ return dataSets.stream().mapToLong(ds -> ds.data.size()).sum();
+ }
+ }
+
+ private static class DataSet {
+ private final List<Mutation> data;
+
+ private int activeScans = 0;
+
+ private boolean deleting = false;
+
+ public DataSet(List<Mutation> data) {
+ this.data = data;
+ }
+
+ synchronized boolean reserveForScan() {
+ if (deleting) {
+ return false;
+ }
+
+ activeScans++;
+
+ return true;
+ }
+
+ synchronized void unreserveForScan() {
+ activeScans--;
+ Preconditions.checkState(activeScans >= 0);
+ if (activeScans == 0) {
+ notify();
+ }
+ }
+
+ synchronized void reserveForDelete() {
+ Preconditions.checkState(!deleting);
+ deleting = true;
+ while (activeScans > 0) {
+ try {
+ wait(50);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ Stream<Key> getExpectedData(Range range) {
+ return data
+ .stream().flatMap(m -> m.getUpdates().stream().map(cu -> new Key(m.getRow(),
+ cu.getColumnFamily(), cu.getColumnQualifier(), cu.getColumnVisibility(), 0L)))
+ .filter(range::contains);
+ }
+
+ }
+
+ private static class ExpectedScanData implements AutoCloseable {
+
+ private final List<DataSet> reservedData;
+
+ public ExpectedScanData(List<DataSet> reservedData) {
+ this.reservedData = reservedData;
+ }
+
+ /**
+ * @return keys that are expected to exist in the accumulo table
+ */
+ Stream<Key> getExpectedData(Range range) {
+ return reservedData.stream().flatMap(ds -> ds.getExpectedData(range));
+ }
+
+ @Override
+ public void close() {
+ reservedData.forEach(DataSet::unreserveForScan);
+ }
+ }
+
+ private static class TestContext {
+ final DataTracker dataTracker = new DataTracker();
+ final AccumuloClient client;
+ final String table;
+ final AtomicBoolean keepRunning = new AtomicBoolean(true);
+ final AtomicLong generationCounter = new AtomicLong(0);
+
+ private TestContext(AccumuloClient client, String table) {
+ this.client = client;
+ this.table = table;
+ }
+ }
+
+ private static class ScanStats {
+ long scanned;
+ long verified;
+
+ public void add(ScanStats stats) {
+ scanned += stats.scanned;
+ verified += stats.verified;
+ }
+ }
+
+ private static ScanStats scanData(TestContext tctx, Range range, boolean scanIsolated)
+ throws Exception {
+ ScanStats stats = new ScanStats();
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ Scanner scanner = tctx.client.createScanner(tctx.table)) {
+ HashSet<Key> expected = new HashSet<>();
+ expectedScanData.getExpectedData(range).forEach(expected::add);
+ scanner.setRange(range);
+
+ Scanner s = scanner;
+ if (scanIsolated) {
+ s = new IsolatedScanner(scanner);
+ }
+
+ for (Map.Entry<Key,Value> entry : s) {
+ stats.scanned++;
+ Key key = entry.getKey();
+ key.setTimestamp(0);
+ if (expected.remove(key)) {
+ stats.verified++;
+ }
+ }
+
+ assertTrue(expected.isEmpty());
+ }
+ return stats;
+ }
+
+ private static class ScanTask implements Callable<ScanStats> {
+
+ private final TestContext tctx;
+
+ private ScanTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public ScanStats call() throws Exception {
+ ScanStats allStats = new ScanStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ Range range;
+ if (random.nextInt(10) == 0) {
+ // 1 in 10 chance of doing a full table scan
+ range = new Range();
+ } else {
+ long start = nextLongAbs(random);
+ long end = nextLongAbs(random);
+
+ while (end <= start) {
+ end = nextLongAbs(random);
+ }
+
+ range = new Range(String.format("%016x", start), String.format("%016x", end));
+ }
+
+ var stats = scanData(tctx, range, random.nextBoolean());
+ allStats.add(stats);
+ }
+
+ return allStats;
+ }
+ }
+
+ private static class WriteStats {
+ long written;
+ long deleted;
+ }
+
+ private static class WriteTask implements Callable<WriteStats> {
+
+ private final TestContext tctx;
+
+ private WriteTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public WriteStats call() throws Exception {
+ WriteStats stats = new WriteStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ // Each row has on average of 50 cols, so 20K rows would be around 100K entries.
+ if (tctx.dataTracker.estimatedRows() > 100_000 / 50) {
+ // There is a good bit of data, so delete some.
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (Mutation m : tctx.dataTracker.getDeletes()) {
+ writer.addMutation(m);
+ stats.deleted += m.getUpdates().size();
+ }
+ }
+
+ } else {
+
+ List<Mutation> dataAdded = new ArrayList<>();
+
+ int rowsToGenerate = random.nextInt(1000);
+
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (int i = 0; i < rowsToGenerate; i++) {
+ Mutation m = generateMutation(random);
+ writer.addMutation(m);
+ stats.written += m.getUpdates().size();
+ dataAdded.add(m);
+ }
Review Comment:
Yeah that is ok, the scans do not care if they see extra data. They just require that they see all data that was successfully written before the scan started.
There is a very last scan done after all the writers have stopped, that scan does not tolerate extra data and should throw an exception if it sees it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1270900925
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
Review Comment:
Opened #3646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#issuecomment-1643786405
This test is a great idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1271115362
##########
core/src/main/java/org/apache/accumulo/core/metadata/schema/DataFileValue.java:
##########
@@ -105,7 +105,7 @@ public int hashCode() {
@Override
public String toString() {
- return size + " " + numEntries;
+ return size + " " + numEntries + " " + time;
Review Comment:
It would be good to improve the toString. I am thinking maybe I should not make this change in 2.1 because I didn't analyze how this toString method was used. I made the change when trying to figure out the problem with bulk importing delete keys. I'll either improve it or not change it after looking into how its used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1269927119
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
+
+ // let the threads know to exit
+ testContext.keepRunning.set(false);
+
+ for (Future<WriteStats> writeTask : writeTasks) {
+ var stats = writeTask.get();
+ log.debug(String.format("Wrote:%,d Deleted:%,d", stats.written, stats.deleted));
+ assertTrue(stats.written > 0);
+ assertTrue(stats.deleted > 0);
+ }
+
+ for (Future<ScanStats> scanTask : scanTasks) {
+ var stats = scanTask.get();
+ log.debug(String.format("Scanned:%,d verified:%,d", stats.scanned, stats.verified));
+ assertTrue(stats.verified > 0);
+ // These scans were running concurrently with writes, so a scan will see more data than what
+ // was written before the scan started.
+ assertTrue(stats.scanned > stats.verified);
+ }
+
+ log.debug(tableOpsTask.get());
+
+ var stats1 = scanData(testContext, new Range(), false);
+ var stats2 = scanData(testContext, new Range(), true);
+ log.debug(
+ String.format("Final scan, scanned:%,d verified:%,d", stats1.scanned, stats1.verified));
+ assertTrue(stats1.verified > 0);
+ // Should see all expected data now that there are no concurrent writes happening
+ assertEquals(stats1.scanned, stats1.verified);
+ assertEquals(stats2.scanned, stats1.scanned);
+ assertEquals(stats2.verified, stats1.verified);
+ } finally {
+ executor.shutdownNow();
+ }
+
+ }
+
+ /**
+ * Tracks what data has been written and deleted in an Accumulo table.
+ */
+ private static class DataTracker {
+ // ConcurrentLinkedQueue was chosen because its safe to iterate over concurrently w/o locking
+ private final Queue<DataSet> dataSets = new ConcurrentLinkedQueue<>();
+
+ /**
+ * Reserves data for scan so that it will not be deleted and returns the data that is expected
+ * to exist in the table at this point.
+ */
+ public ExpectedScanData beginScan() {
+ List<DataSet> reservedData = new ArrayList<>();
+
+ for (var dataSet : dataSets) {
+ if (dataSet.reserveForScan()) {
+ reservedData.add(dataSet);
+ }
+ }
+
+ return new ExpectedScanData(reservedData);
+ }
+
+ /**
+ * add new data that scans should expect to see
+ */
+ public void addExpectedData(List<Mutation> data) {
+ dataSets.add(new DataSet(data));
+ }
+
+ /**
+ * @return data to delete from the table that is not reserved for scans
+ */
+ public Iterable<Mutation> getDeletes() {
+ DataSet dataSet = dataSets.poll();
+
+ if (dataSet == null) {
+ return List.of();
+ }
+
+ dataSet.reserveForDelete();
+
+ return () -> dataSet.data.stream().map(m -> {
+ Mutation delMutation = new Mutation(m.getRow());
+ m.getUpdates()
+ .forEach(cu -> delMutation.putDelete(cu.getColumnFamily(), cu.getColumnQualifier()));
+ return delMutation;
+ }).iterator();
+ }
+
+ public long estimatedRows() {
+ return dataSets.stream().mapToLong(ds -> ds.data.size()).sum();
+ }
+ }
+
+ private static class DataSet {
+ private final List<Mutation> data;
+
+ private int activeScans = 0;
+
+ private boolean deleting = false;
+
+ public DataSet(List<Mutation> data) {
+ this.data = data;
+ }
+
+ synchronized boolean reserveForScan() {
+ if (deleting) {
+ return false;
+ }
+
+ activeScans++;
+
+ return true;
+ }
+
+ synchronized void unreserveForScan() {
+ activeScans--;
+ Preconditions.checkState(activeScans >= 0);
+ if (activeScans == 0) {
+ notify();
+ }
+ }
+
+ synchronized void reserveForDelete() {
+ Preconditions.checkState(!deleting);
+ deleting = true;
+ while (activeScans > 0) {
+ try {
+ wait(50);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ Stream<Key> getExpectedData(Range range) {
+ return data
+ .stream().flatMap(m -> m.getUpdates().stream().map(cu -> new Key(m.getRow(),
+ cu.getColumnFamily(), cu.getColumnQualifier(), cu.getColumnVisibility(), 0L)))
+ .filter(range::contains);
+ }
+
+ }
+
+ private static class ExpectedScanData implements AutoCloseable {
+
+ private final List<DataSet> reservedData;
+
+ public ExpectedScanData(List<DataSet> reservedData) {
+ this.reservedData = reservedData;
+ }
+
+ /**
+ * @return keys that are expected to exist in the accumulo table
+ */
+ Stream<Key> getExpectedData(Range range) {
+ return reservedData.stream().flatMap(ds -> ds.getExpectedData(range));
+ }
+
+ @Override
+ public void close() {
+ reservedData.forEach(DataSet::unreserveForScan);
+ }
+ }
+
+ private static class TestContext {
+ final DataTracker dataTracker = new DataTracker();
+ final AccumuloClient client;
+ final String table;
+ final AtomicBoolean keepRunning = new AtomicBoolean(true);
+ final AtomicLong generationCounter = new AtomicLong(0);
+
+ private TestContext(AccumuloClient client, String table) {
+ this.client = client;
+ this.table = table;
+ }
+ }
+
+ private static class ScanStats {
+ long scanned;
+ long verified;
+
+ public void add(ScanStats stats) {
+ scanned += stats.scanned;
+ verified += stats.verified;
+ }
+ }
+
+ private static ScanStats scanData(TestContext tctx, Range range, boolean scanIsolated)
+ throws Exception {
+ ScanStats stats = new ScanStats();
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ Scanner scanner = tctx.client.createScanner(tctx.table)) {
+ HashSet<Key> expected = new HashSet<>();
+ expectedScanData.getExpectedData(range).forEach(expected::add);
+ scanner.setRange(range);
+
+ Scanner s = scanner;
+ if (scanIsolated) {
+ s = new IsolatedScanner(scanner);
+ }
+
+ for (Map.Entry<Key,Value> entry : s) {
+ stats.scanned++;
+ Key key = entry.getKey();
+ key.setTimestamp(0);
+ if (expected.remove(key)) {
+ stats.verified++;
+ }
+ }
+
+ assertTrue(expected.isEmpty());
+ }
+ return stats;
+ }
+
+ private static class ScanTask implements Callable<ScanStats> {
+
+ private final TestContext tctx;
+
+ private ScanTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public ScanStats call() throws Exception {
+ ScanStats allStats = new ScanStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ Range range;
+ if (random.nextInt(10) == 0) {
+ // 1 in 10 chance of doing a full table scan
+ range = new Range();
+ } else {
+ long start = nextLongAbs(random);
+ long end = nextLongAbs(random);
+
+ while (end <= start) {
+ end = nextLongAbs(random);
+ }
+
+ range = new Range(String.format("%016x", start), String.format("%016x", end));
+ }
+
+ var stats = scanData(tctx, range, random.nextBoolean());
+ allStats.add(stats);
+ }
+
+ return allStats;
+ }
+ }
+
+ private static class WriteStats {
+ long written;
+ long deleted;
+ }
+
+ private static class WriteTask implements Callable<WriteStats> {
+
+ private final TestContext tctx;
+
+ private WriteTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public WriteStats call() throws Exception {
+ WriteStats stats = new WriteStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ // Each row has on average of 50 cols, so 20K rows would be around 100K entries.
+ if (tctx.dataTracker.estimatedRows() > 100_000 / 50) {
+ // There is a good bit of data, so delete some.
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (Mutation m : tctx.dataTracker.getDeletes()) {
+ writer.addMutation(m);
+ stats.deleted += m.getUpdates().size();
+ }
+ }
+
+ } else {
+
+ List<Mutation> dataAdded = new ArrayList<>();
+
+ int rowsToGenerate = random.nextInt(1000);
+
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (int i = 0; i < rowsToGenerate; i++) {
+ Mutation m = generateMutation(random);
+ writer.addMutation(m);
+ stats.written += m.getUpdates().size();
+ dataAdded.add(m);
+ }
+ }
+
+ // Make the data just written visible to scans. Now that the writer is closed scans should
+ // be able to see the data in the table.
+ tctx.dataTracker.addExpectedData(dataAdded);
+ }
+ }
+
+ return stats;
+ }
+
+ private Mutation generateMutation(Random random) {
+
+ int cols = random.nextInt(100) + 1;
+
+ // the generation counter ensures the row is unique
+ String row = String.format("%016x:%016x", nextLongAbs(random),
+ tctx.generationCounter.getAndIncrement());
+
+ Mutation m = new Mutation(row);
+ for (int i = 0; i < cols; i++) {
+ m.put(String.valueOf(random.nextInt(10)), String.format("%04x", random.nextInt(256 * 256)),
+ "");
+ }
+
+ return m;
+ }
+ }
+
+ private static class TableOpsTask implements Callable<String> {
+ private final TestContext tctx;
+
+ private TableOpsTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public String call() throws Exception {
+ int numFlushes = 0;
+ int numCompactions = 0;
+ int numSplits = 0;
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+ Thread.sleep(1000);
+
+ int pick = random.nextInt(100);
+
+ if (pick < 10) {
+ // 1 in 10 chance of flushing
+ tctx.client.tableOperations().flush(tctx.table, null, null, random.nextBoolean());
+ numFlushes++;
+ } else if (pick < 15) {
+ // 1 in 20 chance of compacting
+ tctx.client.tableOperations().compact(tctx.table,
+ new CompactionConfig().setFlush(random.nextBoolean()).setWait(random.nextBoolean()));
+ numCompactions++;
+ } else if (pick < 20) {
+ // 1 in 20 chance of splitting
+ int splitsToAdd = random.nextInt(10);
+ TreeSet<Text> splits = new TreeSet<>();
+
+ for (int i = 0; i < splitsToAdd; i++) {
+ splits.add(new Text(String.format("%016x", nextLongAbs(random))));
+ }
+
+ tctx.client.tableOperations().addSplits(tctx.table, splits);
+ numSplits += splitsToAdd;
+ }
Review Comment:
yeah that would be good to add. maybe bulk import too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#issuecomment-1642851138
Created this IT for testing #3634, however realized its good for 2.1.X and #3573 to stress read patterns the metadata table relies on, so decided to create it in 2.1 and merge it forward to elasticity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1270915879
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
+
+ // let the threads know to exit
+ testContext.keepRunning.set(false);
+
+ for (Future<WriteStats> writeTask : writeTasks) {
+ var stats = writeTask.get();
+ log.debug(String.format("Wrote:%,d Deleted:%,d", stats.written, stats.deleted));
+ assertTrue(stats.written > 0);
+ assertTrue(stats.deleted > 0);
+ }
+
+ for (Future<ScanStats> scanTask : scanTasks) {
+ var stats = scanTask.get();
+ log.debug(String.format("Scanned:%,d verified:%,d", stats.scanned, stats.verified));
+ assertTrue(stats.verified > 0);
+ // These scans were running concurrently with writes, so a scan will see more data than what
+ // was written before the scan started.
+ assertTrue(stats.scanned > stats.verified);
+ }
+
+ log.debug(tableOpsTask.get());
+
+ var stats1 = scanData(testContext, new Range(), false);
+ var stats2 = scanData(testContext, new Range(), true);
+ log.debug(
+ String.format("Final scan, scanned:%,d verified:%,d", stats1.scanned, stats1.verified));
+ assertTrue(stats1.verified > 0);
+ // Should see all expected data now that there are no concurrent writes happening
+ assertEquals(stats1.scanned, stats1.verified);
+ assertEquals(stats2.scanned, stats1.scanned);
+ assertEquals(stats2.verified, stats1.verified);
+ } finally {
+ executor.shutdownNow();
+ }
+
+ }
+
+ /**
+ * Tracks what data has been written and deleted in an Accumulo table.
+ */
+ private static class DataTracker {
+ // ConcurrentLinkedQueue was chosen because its safe to iterate over concurrently w/o locking
+ private final Queue<DataSet> dataSets = new ConcurrentLinkedQueue<>();
+
+ /**
+ * Reserves data for scan so that it will not be deleted and returns the data that is expected
+ * to exist in the table at this point.
+ */
+ public ExpectedScanData beginScan() {
+ List<DataSet> reservedData = new ArrayList<>();
+
+ for (var dataSet : dataSets) {
+ if (dataSet.reserveForScan()) {
+ reservedData.add(dataSet);
+ }
+ }
+
+ return new ExpectedScanData(reservedData);
+ }
+
+ /**
+ * add new data that scans should expect to see
+ */
+ public void addExpectedData(List<Mutation> data) {
+ dataSets.add(new DataSet(data));
+ }
+
+ /**
+ * @return data to delete from the table that is not reserved for scans
+ */
+ public Iterable<Mutation> getDeletes() {
+ DataSet dataSet = dataSets.poll();
+
+ if (dataSet == null) {
+ return List.of();
+ }
+
+ dataSet.reserveForDelete();
+
+ return () -> dataSet.data.stream().map(m -> {
+ Mutation delMutation = new Mutation(m.getRow());
+ m.getUpdates()
+ .forEach(cu -> delMutation.putDelete(cu.getColumnFamily(), cu.getColumnQualifier()));
+ return delMutation;
+ }).iterator();
+ }
+
+ public long estimatedRows() {
+ return dataSets.stream().mapToLong(ds -> ds.data.size()).sum();
+ }
+ }
+
+ private static class DataSet {
+ private final List<Mutation> data;
+
+ private int activeScans = 0;
+
+ private boolean deleting = false;
+
+ public DataSet(List<Mutation> data) {
+ this.data = data;
+ }
+
+ synchronized boolean reserveForScan() {
+ if (deleting) {
+ return false;
+ }
+
+ activeScans++;
+
+ return true;
+ }
+
+ synchronized void unreserveForScan() {
+ activeScans--;
+ Preconditions.checkState(activeScans >= 0);
+ if (activeScans == 0) {
+ notify();
+ }
+ }
+
+ synchronized void reserveForDelete() {
+ Preconditions.checkState(!deleting);
+ deleting = true;
+ while (activeScans > 0) {
+ try {
+ wait(50);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ Stream<Key> getExpectedData(Range range) {
+ return data
+ .stream().flatMap(m -> m.getUpdates().stream().map(cu -> new Key(m.getRow(),
+ cu.getColumnFamily(), cu.getColumnQualifier(), cu.getColumnVisibility(), 0L)))
+ .filter(range::contains);
+ }
+
+ }
+
+ private static class ExpectedScanData implements AutoCloseable {
+
+ private final List<DataSet> reservedData;
+
+ public ExpectedScanData(List<DataSet> reservedData) {
+ this.reservedData = reservedData;
+ }
+
+ /**
+ * @return keys that are expected to exist in the accumulo table
+ */
+ Stream<Key> getExpectedData(Range range) {
+ return reservedData.stream().flatMap(ds -> ds.getExpectedData(range));
+ }
+
+ @Override
+ public void close() {
+ reservedData.forEach(DataSet::unreserveForScan);
+ }
+ }
+
+ private static class TestContext {
+ final DataTracker dataTracker = new DataTracker();
+ final AccumuloClient client;
+ final String table;
+ final AtomicBoolean keepRunning = new AtomicBoolean(true);
+ final AtomicLong generationCounter = new AtomicLong(0);
+
+ private TestContext(AccumuloClient client, String table) {
+ this.client = client;
+ this.table = table;
+ }
+ }
+
+ private static class ScanStats {
+ long scanned;
+ long verified;
+
+ public void add(ScanStats stats) {
+ scanned += stats.scanned;
+ verified += stats.verified;
+ }
+ }
+
+ private static ScanStats scanData(TestContext tctx, Range range, boolean scanIsolated)
+ throws Exception {
+ ScanStats stats = new ScanStats();
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ Scanner scanner = tctx.client.createScanner(tctx.table)) {
+ HashSet<Key> expected = new HashSet<>();
+ expectedScanData.getExpectedData(range).forEach(expected::add);
+ scanner.setRange(range);
+
+ Scanner s = scanner;
+ if (scanIsolated) {
+ s = new IsolatedScanner(scanner);
+ }
+
+ for (Map.Entry<Key,Value> entry : s) {
+ stats.scanned++;
+ Key key = entry.getKey();
+ key.setTimestamp(0);
+ if (expected.remove(key)) {
+ stats.verified++;
+ }
+ }
+
+ assertTrue(expected.isEmpty());
+ }
+ return stats;
+ }
+
+ private static class ScanTask implements Callable<ScanStats> {
+
+ private final TestContext tctx;
+
+ private ScanTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public ScanStats call() throws Exception {
+ ScanStats allStats = new ScanStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ Range range;
+ if (random.nextInt(10) == 0) {
+ // 1 in 10 chance of doing a full table scan
+ range = new Range();
+ } else {
+ long start = nextLongAbs(random);
+ long end = nextLongAbs(random);
+
+ while (end <= start) {
+ end = nextLongAbs(random);
+ }
+
+ range = new Range(String.format("%016x", start), String.format("%016x", end));
+ }
+
+ var stats = scanData(tctx, range, random.nextBoolean());
+ allStats.add(stats);
+ }
+
+ return allStats;
+ }
+ }
+
+ private static class WriteStats {
+ long written;
+ long deleted;
+ }
+
+ private static class WriteTask implements Callable<WriteStats> {
+
+ private final TestContext tctx;
+
+ private WriteTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public WriteStats call() throws Exception {
+ WriteStats stats = new WriteStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ // Each row has on average of 50 cols, so 20K rows would be around 100K entries.
+ if (tctx.dataTracker.estimatedRows() > 100_000 / 50) {
+ // There is a good bit of data, so delete some.
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (Mutation m : tctx.dataTracker.getDeletes()) {
+ writer.addMutation(m);
+ stats.deleted += m.getUpdates().size();
+ }
+ }
+
+ } else {
+
+ List<Mutation> dataAdded = new ArrayList<>();
+
+ int rowsToGenerate = random.nextInt(1000);
+
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (int i = 0; i < rowsToGenerate; i++) {
+ Mutation m = generateMutation(random);
+ writer.addMutation(m);
+ stats.written += m.getUpdates().size();
+ dataAdded.add(m);
+ }
+ }
+
+ // Make the data just written visible to scans. Now that the writer is closed scans should
+ // be able to see the data in the table.
+ tctx.dataTracker.addExpectedData(dataAdded);
+ }
+ }
+
+ return stats;
+ }
+
+ private Mutation generateMutation(Random random) {
+
+ int cols = random.nextInt(100) + 1;
+
+ // the generation counter ensures the row is unique
+ String row = String.format("%016x:%016x", nextLongAbs(random),
+ tctx.generationCounter.getAndIncrement());
+
+ Mutation m = new Mutation(row);
+ for (int i = 0; i < cols; i++) {
+ m.put(String.valueOf(random.nextInt(10)), String.format("%04x", random.nextInt(256 * 256)),
+ "");
+ }
+
+ return m;
+ }
+ }
+
+ private static class TableOpsTask implements Callable<String> {
+ private final TestContext tctx;
+
+ private TableOpsTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public String call() throws Exception {
+ int numFlushes = 0;
+ int numCompactions = 0;
+ int numSplits = 0;
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+ Thread.sleep(1000);
+
+ int pick = random.nextInt(100);
+
+ if (pick < 10) {
+ // 1 in 10 chance of flushing
+ tctx.client.tableOperations().flush(tctx.table, null, null, random.nextBoolean());
+ numFlushes++;
+ } else if (pick < 15) {
+ // 1 in 20 chance of compacting
+ tctx.client.tableOperations().compact(tctx.table,
+ new CompactionConfig().setFlush(random.nextBoolean()).setWait(random.nextBoolean()));
+ numCompactions++;
+ } else if (pick < 20) {
+ // 1 in 20 chance of splitting
+ int splitsToAdd = random.nextInt(10);
+ TreeSet<Text> splits = new TreeSet<>();
+
+ for (int i = 0; i < splitsToAdd; i++) {
+ splits.add(new Text(String.format("%016x", nextLongAbs(random))));
+ }
+
+ tctx.client.tableOperations().addSplits(tctx.table, splits);
+ numSplits += splitsToAdd;
+ }
Review Comment:
added merge and other stuff in 7cbc635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1272794649
##########
core/src/main/java/org/apache/accumulo/core/metadata/schema/DataFileValue.java:
##########
@@ -105,7 +105,7 @@ public int hashCode() {
@Override
public String toString() {
- return size + " " + numEntries;
+ return size + " " + numEntries + " " + time;
Review Comment:
That class is used in a huge number of places. I decided to err on the side of caution and not change its toString rather try to analyze everywhere its used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1269926881
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
Review Comment:
Yeah I think there is value in running it longer, but maybe not in the IT. We definitely want to know if does not work on a short run. I think I will add a main method that allows running it. I think it would be good to have as an IT and something that is easy to run for longer periods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1269346231
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
Review Comment:
Is there any value to letting this run longer?
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
+
+ // let the threads know to exit
+ testContext.keepRunning.set(false);
+
+ for (Future<WriteStats> writeTask : writeTasks) {
+ var stats = writeTask.get();
+ log.debug(String.format("Wrote:%,d Deleted:%,d", stats.written, stats.deleted));
+ assertTrue(stats.written > 0);
+ assertTrue(stats.deleted > 0);
+ }
+
+ for (Future<ScanStats> scanTask : scanTasks) {
+ var stats = scanTask.get();
+ log.debug(String.format("Scanned:%,d verified:%,d", stats.scanned, stats.verified));
+ assertTrue(stats.verified > 0);
+ // These scans were running concurrently with writes, so a scan will see more data than what
+ // was written before the scan started.
+ assertTrue(stats.scanned > stats.verified);
+ }
+
+ log.debug(tableOpsTask.get());
+
+ var stats1 = scanData(testContext, new Range(), false);
+ var stats2 = scanData(testContext, new Range(), true);
+ log.debug(
+ String.format("Final scan, scanned:%,d verified:%,d", stats1.scanned, stats1.verified));
+ assertTrue(stats1.verified > 0);
+ // Should see all expected data now that there are no concurrent writes happening
+ assertEquals(stats1.scanned, stats1.verified);
+ assertEquals(stats2.scanned, stats1.scanned);
+ assertEquals(stats2.verified, stats1.verified);
+ } finally {
+ executor.shutdownNow();
+ }
+
+ }
+
+ /**
+ * Tracks what data has been written and deleted in an Accumulo table.
+ */
+ private static class DataTracker {
+ // ConcurrentLinkedQueue was chosen because its safe to iterate over concurrently w/o locking
+ private final Queue<DataSet> dataSets = new ConcurrentLinkedQueue<>();
+
+ /**
+ * Reserves data for scan so that it will not be deleted and returns the data that is expected
+ * to exist in the table at this point.
+ */
+ public ExpectedScanData beginScan() {
+ List<DataSet> reservedData = new ArrayList<>();
+
+ for (var dataSet : dataSets) {
+ if (dataSet.reserveForScan()) {
+ reservedData.add(dataSet);
+ }
+ }
+
+ return new ExpectedScanData(reservedData);
+ }
+
+ /**
+ * add new data that scans should expect to see
+ */
+ public void addExpectedData(List<Mutation> data) {
+ dataSets.add(new DataSet(data));
+ }
+
+ /**
+ * @return data to delete from the table that is not reserved for scans
+ */
+ public Iterable<Mutation> getDeletes() {
+ DataSet dataSet = dataSets.poll();
+
+ if (dataSet == null) {
+ return List.of();
+ }
+
+ dataSet.reserveForDelete();
+
+ return () -> dataSet.data.stream().map(m -> {
+ Mutation delMutation = new Mutation(m.getRow());
+ m.getUpdates()
+ .forEach(cu -> delMutation.putDelete(cu.getColumnFamily(), cu.getColumnQualifier()));
+ return delMutation;
+ }).iterator();
+ }
+
+ public long estimatedRows() {
+ return dataSets.stream().mapToLong(ds -> ds.data.size()).sum();
+ }
+ }
+
+ private static class DataSet {
+ private final List<Mutation> data;
+
+ private int activeScans = 0;
+
+ private boolean deleting = false;
+
+ public DataSet(List<Mutation> data) {
+ this.data = data;
+ }
+
+ synchronized boolean reserveForScan() {
+ if (deleting) {
+ return false;
+ }
+
+ activeScans++;
+
+ return true;
+ }
+
+ synchronized void unreserveForScan() {
+ activeScans--;
+ Preconditions.checkState(activeScans >= 0);
+ if (activeScans == 0) {
+ notify();
+ }
+ }
+
+ synchronized void reserveForDelete() {
+ Preconditions.checkState(!deleting);
+ deleting = true;
+ while (activeScans > 0) {
+ try {
+ wait(50);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ Stream<Key> getExpectedData(Range range) {
+ return data
+ .stream().flatMap(m -> m.getUpdates().stream().map(cu -> new Key(m.getRow(),
+ cu.getColumnFamily(), cu.getColumnQualifier(), cu.getColumnVisibility(), 0L)))
+ .filter(range::contains);
+ }
+
+ }
+
+ private static class ExpectedScanData implements AutoCloseable {
+
+ private final List<DataSet> reservedData;
+
+ public ExpectedScanData(List<DataSet> reservedData) {
+ this.reservedData = reservedData;
+ }
+
+ /**
+ * @return keys that are expected to exist in the accumulo table
+ */
+ Stream<Key> getExpectedData(Range range) {
+ return reservedData.stream().flatMap(ds -> ds.getExpectedData(range));
+ }
+
+ @Override
+ public void close() {
+ reservedData.forEach(DataSet::unreserveForScan);
+ }
+ }
+
+ private static class TestContext {
+ final DataTracker dataTracker = new DataTracker();
+ final AccumuloClient client;
+ final String table;
+ final AtomicBoolean keepRunning = new AtomicBoolean(true);
+ final AtomicLong generationCounter = new AtomicLong(0);
+
+ private TestContext(AccumuloClient client, String table) {
+ this.client = client;
+ this.table = table;
+ }
+ }
+
+ private static class ScanStats {
+ long scanned;
+ long verified;
+
+ public void add(ScanStats stats) {
+ scanned += stats.scanned;
+ verified += stats.verified;
+ }
+ }
+
+ private static ScanStats scanData(TestContext tctx, Range range, boolean scanIsolated)
+ throws Exception {
+ ScanStats stats = new ScanStats();
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ Scanner scanner = tctx.client.createScanner(tctx.table)) {
+ HashSet<Key> expected = new HashSet<>();
+ expectedScanData.getExpectedData(range).forEach(expected::add);
+ scanner.setRange(range);
+
+ Scanner s = scanner;
+ if (scanIsolated) {
+ s = new IsolatedScanner(scanner);
+ }
+
+ for (Map.Entry<Key,Value> entry : s) {
+ stats.scanned++;
+ Key key = entry.getKey();
+ key.setTimestamp(0);
+ if (expected.remove(key)) {
+ stats.verified++;
+ }
+ }
+
+ assertTrue(expected.isEmpty());
+ }
+ return stats;
+ }
+
+ private static class ScanTask implements Callable<ScanStats> {
+
+ private final TestContext tctx;
+
+ private ScanTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public ScanStats call() throws Exception {
+ ScanStats allStats = new ScanStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ Range range;
+ if (random.nextInt(10) == 0) {
+ // 1 in 10 chance of doing a full table scan
+ range = new Range();
+ } else {
+ long start = nextLongAbs(random);
+ long end = nextLongAbs(random);
+
+ while (end <= start) {
+ end = nextLongAbs(random);
+ }
+
+ range = new Range(String.format("%016x", start), String.format("%016x", end));
+ }
+
+ var stats = scanData(tctx, range, random.nextBoolean());
+ allStats.add(stats);
+ }
+
+ return allStats;
+ }
+ }
+
+ private static class WriteStats {
+ long written;
+ long deleted;
+ }
+
+ private static class WriteTask implements Callable<WriteStats> {
+
+ private final TestContext tctx;
+
+ private WriteTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public WriteStats call() throws Exception {
+ WriteStats stats = new WriteStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ // Each row has on average of 50 cols, so 20K rows would be around 100K entries.
+ if (tctx.dataTracker.estimatedRows() > 100_000 / 50) {
+ // There is a good bit of data, so delete some.
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (Mutation m : tctx.dataTracker.getDeletes()) {
+ writer.addMutation(m);
+ stats.deleted += m.getUpdates().size();
+ }
+ }
+
+ } else {
+
+ List<Mutation> dataAdded = new ArrayList<>();
+
+ int rowsToGenerate = random.nextInt(1000);
+
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (int i = 0; i < rowsToGenerate; i++) {
+ Mutation m = generateMutation(random);
+ writer.addMutation(m);
+ stats.written += m.getUpdates().size();
+ dataAdded.add(m);
+ }
+ }
+
+ // Make the data just written visible to scans. Now that the writer is closed scans should
+ // be able to see the data in the table.
+ tctx.dataTracker.addExpectedData(dataAdded);
+ }
+ }
+
+ return stats;
+ }
+
+ private Mutation generateMutation(Random random) {
+
+ int cols = random.nextInt(100) + 1;
+
+ // the generation counter ensures the row is unique
+ String row = String.format("%016x:%016x", nextLongAbs(random),
+ tctx.generationCounter.getAndIncrement());
+
+ Mutation m = new Mutation(row);
+ for (int i = 0; i < cols; i++) {
+ m.put(String.valueOf(random.nextInt(10)), String.format("%04x", random.nextInt(256 * 256)),
+ "");
+ }
+
+ return m;
+ }
+ }
+
+ private static class TableOpsTask implements Callable<String> {
+ private final TestContext tctx;
+
+ private TableOpsTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public String call() throws Exception {
+ int numFlushes = 0;
+ int numCompactions = 0;
+ int numSplits = 0;
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+ Thread.sleep(1000);
+
+ int pick = random.nextInt(100);
+
+ if (pick < 10) {
+ // 1 in 10 chance of flushing
+ tctx.client.tableOperations().flush(tctx.table, null, null, random.nextBoolean());
+ numFlushes++;
+ } else if (pick < 15) {
+ // 1 in 20 chance of compacting
+ tctx.client.tableOperations().compact(tctx.table,
+ new CompactionConfig().setFlush(random.nextBoolean()).setWait(random.nextBoolean()));
+ numCompactions++;
+ } else if (pick < 20) {
+ // 1 in 20 chance of splitting
+ int splitsToAdd = random.nextInt(10);
+ TreeSet<Text> splits = new TreeSet<>();
+
+ for (int i = 0; i < splitsToAdd; i++) {
+ splits.add(new Text(String.format("%016x", nextLongAbs(random))));
+ }
+
+ tctx.client.tableOperations().addSplits(tctx.table, splits);
+ numSplits += splitsToAdd;
+ }
Review Comment:
should we add merging?
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table);
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
+
+ // let the threads know to exit
+ testContext.keepRunning.set(false);
+
+ for (Future<WriteStats> writeTask : writeTasks) {
+ var stats = writeTask.get();
+ log.debug(String.format("Wrote:%,d Deleted:%,d", stats.written, stats.deleted));
+ assertTrue(stats.written > 0);
+ assertTrue(stats.deleted > 0);
+ }
+
+ for (Future<ScanStats> scanTask : scanTasks) {
+ var stats = scanTask.get();
+ log.debug(String.format("Scanned:%,d verified:%,d", stats.scanned, stats.verified));
+ assertTrue(stats.verified > 0);
+ // These scans were running concurrently with writes, so a scan will see more data than what
+ // was written before the scan started.
+ assertTrue(stats.scanned > stats.verified);
+ }
+
+ log.debug(tableOpsTask.get());
+
+ var stats1 = scanData(testContext, new Range(), false);
+ var stats2 = scanData(testContext, new Range(), true);
+ log.debug(
+ String.format("Final scan, scanned:%,d verified:%,d", stats1.scanned, stats1.verified));
+ assertTrue(stats1.verified > 0);
+ // Should see all expected data now that there are no concurrent writes happening
+ assertEquals(stats1.scanned, stats1.verified);
+ assertEquals(stats2.scanned, stats1.scanned);
+ assertEquals(stats2.verified, stats1.verified);
+ } finally {
+ executor.shutdownNow();
+ }
+
+ }
+
+ /**
+ * Tracks what data has been written and deleted in an Accumulo table.
+ */
+ private static class DataTracker {
+ // ConcurrentLinkedQueue was chosen because its safe to iterate over concurrently w/o locking
+ private final Queue<DataSet> dataSets = new ConcurrentLinkedQueue<>();
+
+ /**
+ * Reserves data for scan so that it will not be deleted and returns the data that is expected
+ * to exist in the table at this point.
+ */
+ public ExpectedScanData beginScan() {
+ List<DataSet> reservedData = new ArrayList<>();
+
+ for (var dataSet : dataSets) {
+ if (dataSet.reserveForScan()) {
+ reservedData.add(dataSet);
+ }
+ }
+
+ return new ExpectedScanData(reservedData);
+ }
+
+ /**
+ * add new data that scans should expect to see
+ */
+ public void addExpectedData(List<Mutation> data) {
+ dataSets.add(new DataSet(data));
+ }
+
+ /**
+ * @return data to delete from the table that is not reserved for scans
+ */
+ public Iterable<Mutation> getDeletes() {
+ DataSet dataSet = dataSets.poll();
+
+ if (dataSet == null) {
+ return List.of();
+ }
+
+ dataSet.reserveForDelete();
+
+ return () -> dataSet.data.stream().map(m -> {
+ Mutation delMutation = new Mutation(m.getRow());
+ m.getUpdates()
+ .forEach(cu -> delMutation.putDelete(cu.getColumnFamily(), cu.getColumnQualifier()));
+ return delMutation;
+ }).iterator();
+ }
+
+ public long estimatedRows() {
+ return dataSets.stream().mapToLong(ds -> ds.data.size()).sum();
+ }
+ }
+
+ private static class DataSet {
+ private final List<Mutation> data;
+
+ private int activeScans = 0;
+
+ private boolean deleting = false;
+
+ public DataSet(List<Mutation> data) {
+ this.data = data;
+ }
+
+ synchronized boolean reserveForScan() {
+ if (deleting) {
+ return false;
+ }
+
+ activeScans++;
+
+ return true;
+ }
+
+ synchronized void unreserveForScan() {
+ activeScans--;
+ Preconditions.checkState(activeScans >= 0);
+ if (activeScans == 0) {
+ notify();
+ }
+ }
+
+ synchronized void reserveForDelete() {
+ Preconditions.checkState(!deleting);
+ deleting = true;
+ while (activeScans > 0) {
+ try {
+ wait(50);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ Stream<Key> getExpectedData(Range range) {
+ return data
+ .stream().flatMap(m -> m.getUpdates().stream().map(cu -> new Key(m.getRow(),
+ cu.getColumnFamily(), cu.getColumnQualifier(), cu.getColumnVisibility(), 0L)))
+ .filter(range::contains);
+ }
+
+ }
+
+ private static class ExpectedScanData implements AutoCloseable {
+
+ private final List<DataSet> reservedData;
+
+ public ExpectedScanData(List<DataSet> reservedData) {
+ this.reservedData = reservedData;
+ }
+
+ /**
+ * @return keys that are expected to exist in the accumulo table
+ */
+ Stream<Key> getExpectedData(Range range) {
+ return reservedData.stream().flatMap(ds -> ds.getExpectedData(range));
+ }
+
+ @Override
+ public void close() {
+ reservedData.forEach(DataSet::unreserveForScan);
+ }
+ }
+
+ private static class TestContext {
+ final DataTracker dataTracker = new DataTracker();
+ final AccumuloClient client;
+ final String table;
+ final AtomicBoolean keepRunning = new AtomicBoolean(true);
+ final AtomicLong generationCounter = new AtomicLong(0);
+
+ private TestContext(AccumuloClient client, String table) {
+ this.client = client;
+ this.table = table;
+ }
+ }
+
+ private static class ScanStats {
+ long scanned;
+ long verified;
+
+ public void add(ScanStats stats) {
+ scanned += stats.scanned;
+ verified += stats.verified;
+ }
+ }
+
+ private static ScanStats scanData(TestContext tctx, Range range, boolean scanIsolated)
+ throws Exception {
+ ScanStats stats = new ScanStats();
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ Scanner scanner = tctx.client.createScanner(tctx.table)) {
+ HashSet<Key> expected = new HashSet<>();
+ expectedScanData.getExpectedData(range).forEach(expected::add);
+ scanner.setRange(range);
+
+ Scanner s = scanner;
+ if (scanIsolated) {
+ s = new IsolatedScanner(scanner);
+ }
+
+ for (Map.Entry<Key,Value> entry : s) {
+ stats.scanned++;
+ Key key = entry.getKey();
+ key.setTimestamp(0);
+ if (expected.remove(key)) {
+ stats.verified++;
+ }
+ }
+
+ assertTrue(expected.isEmpty());
+ }
+ return stats;
+ }
+
+ private static class ScanTask implements Callable<ScanStats> {
+
+ private final TestContext tctx;
+
+ private ScanTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public ScanStats call() throws Exception {
+ ScanStats allStats = new ScanStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ Range range;
+ if (random.nextInt(10) == 0) {
+ // 1 in 10 chance of doing a full table scan
+ range = new Range();
+ } else {
+ long start = nextLongAbs(random);
+ long end = nextLongAbs(random);
+
+ while (end <= start) {
+ end = nextLongAbs(random);
+ }
+
+ range = new Range(String.format("%016x", start), String.format("%016x", end));
+ }
+
+ var stats = scanData(tctx, range, random.nextBoolean());
+ allStats.add(stats);
+ }
+
+ return allStats;
+ }
+ }
+
+ private static class WriteStats {
+ long written;
+ long deleted;
+ }
+
+ private static class WriteTask implements Callable<WriteStats> {
+
+ private final TestContext tctx;
+
+ private WriteTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public WriteStats call() throws Exception {
+ WriteStats stats = new WriteStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ // Each row has on average of 50 cols, so 20K rows would be around 100K entries.
+ if (tctx.dataTracker.estimatedRows() > 100_000 / 50) {
+ // There is a good bit of data, so delete some.
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (Mutation m : tctx.dataTracker.getDeletes()) {
+ writer.addMutation(m);
+ stats.deleted += m.getUpdates().size();
+ }
+ }
+
+ } else {
+
+ List<Mutation> dataAdded = new ArrayList<>();
+
+ int rowsToGenerate = random.nextInt(1000);
+
+ try (var writer = tctx.client.createBatchWriter(tctx.table)) {
+ for (int i = 0; i < rowsToGenerate; i++) {
+ Mutation m = generateMutation(random);
+ writer.addMutation(m);
+ stats.written += m.getUpdates().size();
+ dataAdded.add(m);
+ }
Review Comment:
Is it possible that the BatchWriter flushes in this loop based on default configs? If so, a scan that starts during this loop will see more data than expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a diff in pull request #3639: Adds IT that verifies scans see data written by concurrent writers
Posted by "EdColeman (via GitHub)" <gi...@apache.org>.
EdColeman commented on code in PR #3639:
URL: https://github.com/apache/accumulo/pull/3639#discussion_r1270896502
##########
test/src/main/java/org/apache/accumulo/test/ScanConsistencyIT.java:
##########
@@ -0,0 +1,697 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.harness.AccumuloITBase.SUNNY_DAY;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Random;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.BatchScanner;
+import org.apache.accumulo.core.client.IsolatedScanner;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.Scanner;
+import org.apache.accumulo.core.client.ScannerBase;
+import org.apache.accumulo.core.client.admin.CompactionConfig;
+import org.apache.accumulo.core.client.rfile.RFile;
+import org.apache.accumulo.core.client.rfile.RFileWriter;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorEnvironment;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.harness.AccumuloClusterHarness;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Collections2;
+import com.google.common.collect.MoreCollectors;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+
+/**
+ * This test verifies that scans will always see data written before the scan started even when
+ * there are concurrent scans, writes, and table operations running.
+ */
+@Tag(SUNNY_DAY)
+public class ScanConsistencyIT extends AccumuloClusterHarness {
+
+ private static final Logger log = LoggerFactory.getLogger(ScanConsistencyIT.class);
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Test
+ public void testConcurrentScanConsistency() throws Exception {
+ final String table = this.getUniqueNames(1)[0];
+
+ /**
+ * Tips for debugging this test when it sees a row that should not exist or does not see a row
+ * that should exist.
+ *
+ * 1. Disable the GC from running for the test.
+ *
+ * 2. Modify the test code to print some of the offending rows, just need a few to start
+ * investigating.
+ *
+ * 3. After the test fails, somehow run the static function findRow() passing it the Accumulo
+ * table directory that the failed test used and one of the problem rows.
+ *
+ * 4. Once the files containing the row is found, analyze what happened with those files in the
+ * servers logs.
+ */
+ // getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+
+ var executor = Executors.newCachedThreadPool();
+ try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
+ client.tableOperations().create(table);
+
+ TestContext testContext = new TestContext(client, table, getCluster().getFileSystem(),
+ getCluster().getTemporaryPath().toString());
+
+ List<Future<WriteStats>> writeTasks = new ArrayList<>();
+ List<Future<ScanStats>> scanTasks = new ArrayList<>();
+
+ Random random = new Random();
+
+ int numWriteTask = random.nextInt(10) + 1;
+ int numsScanTask = random.nextInt(10) + 1;
+
+ for (int i = 0; i < numWriteTask; i++) {
+ writeTasks.add(executor.submit(new WriteTask(testContext)));
+ }
+
+ for (int i = 0; i < numsScanTask; i++) {
+ scanTasks.add(executor.submit(new ScanTask(testContext)));
+ }
+
+ var tableOpsTask = executor.submit(new TableOpsTask(testContext));
+
+ // let the concurrent mayhem run for a bit
+ Thread.sleep(60000);
+
+ // let the threads know to exit
+ testContext.keepRunning.set(false);
+
+ for (Future<WriteStats> writeTask : writeTasks) {
+ var stats = writeTask.get();
+ log.debug(String.format("Wrote:%,d Bulk imported:%,d Deleted:%,d Bulk deleted:%,d",
+ stats.written, stats.bulkImported, stats.deleted, stats.bulkDeleted));
+ assertTrue(stats.written + stats.bulkImported > 0);
+ assertTrue(stats.deleted + stats.bulkDeleted > 0);
+ }
+
+ for (Future<ScanStats> scanTask : scanTasks) {
+ var stats = scanTask.get();
+ log.debug(String.format("Scanned:%,d verified:%,d", stats.scanned, stats.verified));
+ assertTrue(stats.verified > 0);
+ // These scans were running concurrently with writes, so a scan will see more data than what
+ // was written before the scan started.
+ assertTrue(stats.scanned > stats.verified);
+ }
+
+ log.debug(tableOpsTask.get());
+
+ var stats1 = scanData(testContext, new Range(), false);
+ var stats2 = scanData(testContext, new Range(), true);
+ log.debug(
+ String.format("Final scan, scanned:%,d verified:%,d", stats1.scanned, stats1.verified));
+ assertTrue(stats1.verified > 0);
+ // Should see all expected data now that there are no concurrent writes happening
+ assertEquals(stats1.scanned, stats1.verified);
+ assertEquals(stats2.scanned, stats1.scanned);
+ assertEquals(stats2.verified, stats1.verified);
+ } finally {
+ executor.shutdownNow();
+ }
+ }
+
+ /**
+ * Tracks what data has been written and deleted in an Accumulo table.
+ */
+ private static class DataTracker {
+ // ConcurrentLinkedQueue was chosen because its safe to iterate over concurrently w/o locking
+ private final Queue<DataSet> dataSets = new ConcurrentLinkedQueue<>();
+
+ /**
+ * Reserves data for scan so that it will not be deleted and returns the data that is expected
+ * to exist in the table at this point.
+ */
+ public ExpectedScanData beginScan() {
+ List<DataSet> reservedData = new ArrayList<>();
+
+ for (var dataSet : dataSets) {
+ if (dataSet.reserveForScan()) {
+ reservedData.add(dataSet);
+ }
+ }
+
+ return new ExpectedScanData(reservedData);
+ }
+
+ /**
+ * add new data that scans should expect to see
+ */
+ public void addExpectedData(List<Mutation> data) {
+ dataSets.add(new DataSet(data));
+ }
+
+ /**
+ * @return data to delete from the table that is not reserved for scans
+ */
+ public Collection<Mutation> getDeletes() {
+ DataSet dataSet = dataSets.poll();
+
+ if (dataSet == null) {
+ return List.of();
+ }
+
+ dataSet.reserveForDelete();
+
+ return Collections2.transform(dataSet.data, m -> {
+ Mutation delMutation = new Mutation(m.getRow());
+ m.getUpdates()
+ .forEach(cu -> delMutation.putDelete(cu.getColumnFamily(), cu.getColumnQualifier()));
+ return delMutation;
+ });
+ }
+
+ public long estimatedRows() {
+ return dataSets.stream().mapToLong(ds -> ds.data.size()).sum();
+ }
+ }
+
+ private static class DataSet {
+ private final List<Mutation> data;
+
+ private int activeScans = 0;
+
+ private boolean deleting = false;
+
+ public DataSet(List<Mutation> data) {
+ this.data = data;
+ }
+
+ synchronized boolean reserveForScan() {
+ if (deleting) {
+ return false;
+ }
+
+ activeScans++;
+
+ return true;
+ }
+
+ synchronized void unreserveForScan() {
+ activeScans--;
+ Preconditions.checkState(activeScans >= 0);
+ if (activeScans == 0) {
+ notify();
+ }
+ }
+
+ synchronized void reserveForDelete() {
+ Preconditions.checkState(!deleting);
+ deleting = true;
+ while (activeScans > 0) {
+ try {
+ wait(50);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ Stream<Key> getExpectedData(Range range) {
+ return data.stream().flatMap(ScanConsistencyIT::toKeys).filter(range::contains);
+ }
+ }
+
+ private static Stream<Key> toKeys(Mutation m) {
+ return m.getUpdates().stream().map(cu -> new Key(m.getRow(), cu.getColumnFamily(),
+ cu.getColumnQualifier(), cu.getColumnVisibility(), 0L, cu.isDeleted(), false));
+ }
+
+ private static class ExpectedScanData implements AutoCloseable {
+
+ private final List<DataSet> reservedData;
+
+ public ExpectedScanData(List<DataSet> reservedData) {
+ this.reservedData = reservedData;
+ }
+
+ /**
+ * @return keys that are expected to exist in the accumulo table
+ */
+ Stream<Key> getExpectedData(Range range) {
+ return reservedData.stream().flatMap(ds -> ds.getExpectedData(range));
+ }
+
+ @Override
+ public void close() {
+ reservedData.forEach(DataSet::unreserveForScan);
+ }
+ }
+
+ private static class TestContext {
+ final DataTracker dataTracker = new DataTracker();
+ final AccumuloClient client;
+ final String table;
+ final AtomicBoolean keepRunning = new AtomicBoolean(true);
+ final AtomicLong generationCounter = new AtomicLong(0);
+ final FileSystem fileSystem;
+ private final String tmpDir;
+
+ private TestContext(AccumuloClient client, String table, FileSystem fs, String tmpDir) {
+ this.client = client;
+ this.table = table;
+ this.fileSystem = fs;
+ this.tmpDir = tmpDir;
+ }
+ }
+
+ private static class ScanStats {
+ long scanned;
+ long verified;
+
+ public void add(ScanStats stats) {
+ scanned += stats.scanned;
+ verified += stats.verified;
+ }
+ }
+
+ private static ScanStats scan(ScannerBase scanner, Set<Key> expected) {
+ ScanStats stats = new ScanStats();
+ for (Map.Entry<Key,Value> entry : scanner) {
+ stats.scanned++;
+ Key key = entry.getKey();
+ key.setTimestamp(0);
+ if (expected.remove(key)) {
+ stats.verified++;
+ }
+ }
+
+ assertTrue(expected.isEmpty());
+ return stats;
+ }
+
+ // TODO create multiple ranges for batch scanner
+ private static ScanStats batchScanData(TestContext tctx, Range range) throws Exception {
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ BatchScanner scanner = tctx.client.createBatchScanner(tctx.table)) {
+ Set<Key> expected = expectedScanData.getExpectedData(range).collect(Collectors.toSet());
+ scanner.setRanges(List.of(range));
+ return scan(scanner, expected);
+ }
+ }
+
+ private static ScanStats scanData(TestContext tctx, Range range, boolean scanIsolated)
+ throws Exception {
+ try (ExpectedScanData expectedScanData = tctx.dataTracker.beginScan();
+ Scanner scanner = tctx.client.createScanner(tctx.table)) {
+ Set<Key> expected = expectedScanData.getExpectedData(range).collect(Collectors.toSet());
+ scanner.setRange(range);
+
+ Scanner s = scanner;
+ if (scanIsolated) {
+ s = new IsolatedScanner(scanner);
+ }
+
+ return scan(s, expected);
+ }
+ }
+
+ private static class ScanTask implements Callable<ScanStats> {
+
+ private final TestContext tctx;
+
+ private ScanTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ @SuppressFBWarnings(value = {"PREDICTABLE_RANDOM", "DMI_RANDOM_USED_ONLY_ONCE"},
+ justification = "predictable random is ok for testing")
+ @Override
+ public ScanStats call() throws Exception {
+ ScanStats allStats = new ScanStats();
+
+ Random random = new Random();
+
+ while (tctx.keepRunning.get()) {
+
+ Range range;
+ if (random.nextInt(10) == 0) {
+ // 1 in 10 chance of doing a full table scan
+ range = new Range();
+ } else {
+ long start = nextLongAbs(random);
+ long end = nextLongAbs(random);
+
+ while (end <= start) {
+ end = nextLongAbs(random);
+ }
+
+ range = new Range(String.format("%016x", start), String.format("%016x", end));
+ }
+
+ int scanChance = random.nextInt(3);
+ if (scanChance == 0) {
+ allStats.add(scanData(tctx, range, false));
+ } else if (scanChance == 1) {
+ allStats.add(scanData(tctx, range, true));
+ } else {
+ allStats.add(batchScanData(tctx, range));
+ }
+ }
+
+ return allStats;
+ }
+ }
+
+ private static class WriteStats {
+ long written;
+ long deleted;
+ long bulkImported;
+ long bulkDeleted;
+ }
+
+ private static class WriteTask implements Callable<WriteStats> {
+
+ private final TestContext tctx;
+
+ private WriteTask(TestContext testContext) {
+ this.tctx = testContext;
+ }
+
+ private long bulkImport(Random random, Collection<Mutation> mutations) throws Exception {
+
+ if (mutations.isEmpty()) {
+ return 0;
+ }
+
+ Path bulkDir = new Path(tctx.tmpDir + "/" + "bulkimport_" + nextLongAbs(random));
Review Comment:
```suggestion
Path bulkDir = new Path(tctx.tmpDir + "/bulkimport_" + nextLongAbs(random));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org