You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "kevinrr888 (via GitHub)" <gi...@apache.org> on 2025/04/14 15:56:43 UTC
[PR] Comprehensive table operations IT [accumulo]
kevinrr888 opened a new pull request, #5474:
URL: https://github.com/apache/accumulo/pull/5474
This PR:
Creates a comprehensive IT of all table operations against both system and user tables while trying to avoid duplicating existing testing. For example, when there was more extensive testing of a table operation elsewhere, these new tests just ensure that that testing still exists. This IT does not test for edge cases, but rather tests for basic expected functionality of all table operations against user tables and all system tables (ROOT, METADATA, FATE, SCAN_REF).
Motivation:
- No single test testing all table operations making it hard to know what is or isn't tested
- Lots of operations that aren't tested on all system tables and some operations that are completely untested. Good to ensure each table op works as expected on the different system tables as well as user tables. For example, some ops can't be performed on the metadata tables (ROOT, METADATA) but can be performed on FATE and SCAN_REF, some ops aren't allowed on any system tables, etc.
- Has made identifying potential issues related to what ops are allowed on what system tables easier. For example, have discovered that `offline`ing the FATE and SCAN_REF tables are allowed through the table operations API when they shouldn't be (#5433)
- Ensures end-to-end functionality of table operations always work as expected
- Ensures FATE, as it's used in the Manager, is functioning as expected for both META (FATE ops against system tables) and USER (FATE ops against user tables) FATE ops. This is important since they use different resources and different stores. Lots of individual FATE tests, but this verifies FATE is working as expected in the Manager.
- Will force any newly added table operations to have testing here.
- Will force any new system tables to be considered in these tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050989517
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
Review Comment:
Since this is already tested in `test_list_tableIdMap`, chose to avoid same checks elsewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2051102793
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
Review Comment:
Thanks, yeah, should be setting userTable to null here to prevent accidentally reusing the same userTable across tests as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060499303
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,937 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TabletMergeability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.clientImpl.TabletMergeabilityUtil;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ userTable = null;
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps =
+ Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName);
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var untestedOps =
+ allTableOps.filter(op -> testMethodNames.stream().noneMatch(test -> test.contains(op)))
+ .collect(Collectors.toSet());
+ assertTrue(untestedOps.isEmpty(), "The table operations " + untestedOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ for (Map.Entry<String,String> entry : ops.tableIdMap().entrySet()) {
+ var tableName = entry.getKey();
+ var tableId = TableId.of(entry.getValue());
+ if (tableName.equals(AccumuloTable.ROOT.tableName())) {
+ assertEquals(AccumuloTable.ROOT.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.METADATA.tableName())) {
+ assertEquals(AccumuloTable.METADATA.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.FATE.tableName())) {
+ assertEquals(AccumuloTable.FATE.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.SCAN_REF.tableName())) {
+ assertEquals(AccumuloTable.SCAN_REF.tableId(), tableId);
+ } else if (tableName.equals(userTable)) {
+ assertFalse(AccumuloTable.allTableIds().contains(tableId));
+ } else {
+ throw new IllegalStateException("Unrecognized table: " + tableName);
+ }
+ }
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // addSplits,listSplits,putSplits tested elsewhere for METADATA, ROOT, and user tables, but
+ // testing here as well since this setup is needed to test for splitRangeByTablets anyway,
+ // which is untested elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = ops.list();
+
+ for (String table : allTables) {
+ SortedSet<Text> splits1 = new TreeSet<>();
+ splits1.add(new Text("split1"));
+ SortedSet<Text> splits2 = new TreeSet<>();
+ splits2.add(new Text("split2"));
+ SortedMap<Text,TabletMergeability> splits2Map =
+ TabletMergeabilityUtil.userDefaultSplits(splits2);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits1));
+ assertThrows(AccumuloException.class, () -> ops.putSplits(table, splits2Map));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits1);
+ ops.putSplits(table, splits2Map);
+ var listSplits = ops.listSplits(table);
+ assertTrue(listSplits.containsAll(splits1));
+ assertTrue(listSplits.containsAll(splits2));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class, () -> ops.rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
Review Comment:
Added in c88263153459d77dad6ce76bb3c7ac4449bdbde1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060372742
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
Review Comment:
That would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2052576349
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
Review Comment:
Created https://github.com/apache/accumulo/pull/5491
When testing this, noticed you can call `client.tableOperations.deleteRows` with no problems for FATE and SCAN_REF, but trying to use a batch writer
(`BatchWriter bw = client.createBatchWriter(AccumuloTable.SCAN_REF.tableName())` for example) you cannot write mutations this way:
`org.apache.accumulo.core.client.MutationsRejectedException: constraint violation codes : [] security codes: {accumulo.scanref(ID:+scanref)=[PERMISSION_DENIED]} # server errors 0 # exceptions 0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060458180
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
Review Comment:
deleteRows on system tables is no longer possible. Test now expects deleteRows to throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054324888
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
Review Comment:
Thats ok, its similar to other comments. That is a convenience method for setting table props. So as long as the user has that permission it should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050990073
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
Review Comment:
Yes, good point. Added testing for `putSplits` in 4eb99ebd482139c834dbad28948a1842ab88b7d5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054320462
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
Review Comment:
It should be allowed as long as the user has permissions to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2047804617
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
Review Comment:
Would be nice to check the entire map here, including the values. For system tables we know what the id should be for a name. For the user table its fine being anything that is not a system table id.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
Review Comment:
Making assumptions about the impl in the test will not catch regressions. Seems like it would be best to test both.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
Review Comment:
This test is really neat.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
Review Comment:
Neat way of checking this assumption.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
Review Comment:
Could check the size of `allTables`, but maybe that is redundant w/ other testing of list.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
Review Comment:
This is a nice finding, should not be able to offline the new fate table.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
Review Comment:
Seems like it would be ok to add a hard coded check to FateServiceHandler for this like the root and metadata table.
Kinda related, wondering if we have any test that check the default out of the box permissions of the system tables. Looking into this question I saw that calling this operation requires the write permission on a tablet. What this operation does can be done much slower w/ a batch writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050993271
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
Review Comment:
#5433 has been merged in. This test has been updated (in 4eb99ebd482139c834dbad28948a1842ab88b7d5) to expect an exception when offlining any system table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2167183864
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
Review Comment:
Opened https://github.com/apache/accumulo/issues/5679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054322830
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
Review Comment:
For these different exceptions you are seeing, do they seem correct? Like is the AccumuloException related to a not security issue and it the AccumuloSecurityException related to a security issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050995597
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
Review Comment:
https://github.com/apache/accumulo/pull/5479 fixed this issue and 4eb99ebd482139c834dbad28948a1842ab88b7d5 removed this skip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060456719
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
+ assertTrue(ops.getTabletInformation(sysTable.tableName(), new Range()).findAny().isPresent());
+ ops.getTabletInformation(sysTable.tableName(), new Range())
+ .forEach(ti -> assertEquals(TabletAvailability.HOSTED, ti.getTabletAvailability()));
+ }
+ }
+
+ /**
+ * Creates some data in the FATE table. Will create a single row each call. Does so by initiating
+ * a very slow compaction on the given table.
+ *
+ * @param table the table the fate operation will operate on
+ * @return the row created
+ */
+ private Text createFateTableRow(String table) throws Exception {
+ attachSlowMajcIterator(table);
+ ReadWriteIT.ingest(client, 5, 5, 5, 0, table);
+
+ Set<Text> rowsSeenBeforeNewOp = new HashSet<>();
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ for (var entry : scanner) {
+ rowsSeenBeforeNewOp.add(entry.getKey().getRow());
+ }
+ }
+
+ // start a very slow compaction to create a FATE op that will linger in the FATE table until
+ // cancelled
+ client.tableOperations().compact(table, null, null, true, false);
+
+ Set<Text> rowsSeenAfterNewOp = new HashSet<>();
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ for (var entry : scanner) {
+ rowsSeenAfterNewOp.add(entry.getKey().getRow());
+ }
+ }
+
+ var newOp = Sets.difference(rowsSeenAfterNewOp, rowsSeenBeforeNewOp);
+ assertEquals(1, newOp.size());
+ return newOp.stream().findFirst().orElseThrow();
+ }
+
+ private void attachSlowMajcIterator(String table) throws Exception {
+ if (!ops.listIterators(table).containsKey(SLOW_ITER_NAME)) {
+ IteratorSetting is = new IteratorSetting(1, SLOW_ITER_NAME, SlowIterator.class);
+ is.addOption("sleepTime", "60000");
+ ops.attachIterator(table, is, EnumSet.of(IteratorUtil.IteratorScope.majc));
+ }
+ }
+
+ private void removeSlowMajcIterator(String table) throws Exception {
+ ops.removeIterator(table, SLOW_ITER_NAME, EnumSet.of(IteratorUtil.IteratorScope.majc));
+ }
+
+ /**
+ * Creates some data in the SCAN_REF table. Will create a single row each call.
+ *
+ * @return the row created
+ */
+ private Text createScanRefTableRow() {
+ HostAndPort server = HostAndPort.fromParts("127.0.0.1", 1234);
+ UUID serverLockUUID = UUID.randomUUID();
+ Set<ScanServerRefTabletFile> scanRefs = Stream.of("F0000070.rf", "F0000071.rf")
+ .map(f -> "hdfs://localhost:8020/accumulo/tables/2a/default_tablet/" + f)
+ .map(f -> new ScanServerRefTabletFile(f, server.toString(), serverLockUUID))
+ .collect(Collectors.toSet());
+ getCluster().getServerContext().getAmple().scanServerRefs().put(scanRefs);
+ return new Text(getCluster().getServerContext().getAmple().scanServerRefs().list()
+ .filter(tf -> tf.getServerLockUUID().equals(serverLockUUID)).findFirst().orElseThrow()
+ .getServerLockUUID().toString());
+ }
+
+ /**
+ * Cleans up the data in the FATE table that was created by calls to
+ * {@link #createFateTableRow(String)}
+ */
+ private void cleanupFateTable() throws Exception {
+ ops.cancelCompaction(userTable);
+ // Wait for FATE table to be clear
+ Wait.waitFor(() -> {
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ return !scanner.iterator().hasNext();
+ }
+ });
+ }
+
+ /**
+ * Cleans up the data in the SCAN_REF table that was created by calls to
+ * {@link #createScanRefTableRow()}
+ */
+ private void cleanupScanRefTable() throws Exception {
+ ops.deleteRows(AccumuloTable.SCAN_REF.tableName(), null, null);
Review Comment:
Replaced deleteRows here. deleteRows on system tables is no longer possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060550506
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
Review Comment:
Verified that permissions are checked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2049417316
##########
core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java:
##########
@@ -660,6 +660,7 @@ private void requestTabletHosting(ClientContext context,
}
// System tables should always be hosted
+ // TODO should this return for all sys tables?
if (AccumuloTable.ROOT.tableId() == tableId || AccumuloTable.METADATA.tableId() == tableId) {
return;
}
Review Comment:
Created PR #5486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050994536
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
Review Comment:
#5433 has been merged in. This test has been updated (in 4eb99ebd482139c834dbad28948a1842ab88b7d5) to expect an exception when offlining any system table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2051103128
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
Review Comment:
Certainly much more concise. Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2167197874
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
Review Comment:
Functionality fixed in #5510, assertion fixed in 937e4f8c595b128e94a32ab0e2b687a0ff8cfe31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050817134
##########
core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java:
##########
@@ -660,6 +660,7 @@ private void requestTabletHosting(ClientContext context,
}
// System tables should always be hosted
+ // TODO should this return for all sys tables?
if (AccumuloTable.ROOT.tableId() == tableId || AccumuloTable.METADATA.tableId() == tableId) {
return;
}
Review Comment:
Created https://github.com/apache/accumulo/pull/5487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054390868
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,937 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TabletMergeability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.clientImpl.TabletMergeabilityUtil;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ userTable = null;
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps =
+ Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName);
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var untestedOps =
+ allTableOps.filter(op -> testMethodNames.stream().noneMatch(test -> test.contains(op)))
+ .collect(Collectors.toSet());
+ assertTrue(untestedOps.isEmpty(), "The table operations " + untestedOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ for (Map.Entry<String,String> entry : ops.tableIdMap().entrySet()) {
+ var tableName = entry.getKey();
+ var tableId = TableId.of(entry.getValue());
+ if (tableName.equals(AccumuloTable.ROOT.tableName())) {
+ assertEquals(AccumuloTable.ROOT.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.METADATA.tableName())) {
+ assertEquals(AccumuloTable.METADATA.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.FATE.tableName())) {
+ assertEquals(AccumuloTable.FATE.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.SCAN_REF.tableName())) {
+ assertEquals(AccumuloTable.SCAN_REF.tableId(), tableId);
+ } else if (tableName.equals(userTable)) {
+ assertFalse(AccumuloTable.allTableIds().contains(tableId));
+ } else {
+ throw new IllegalStateException("Unrecognized table: " + tableName);
+ }
+ }
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // addSplits,listSplits,putSplits tested elsewhere for METADATA, ROOT, and user tables, but
+ // testing here as well since this setup is needed to test for splitRangeByTablets anyway,
+ // which is untested elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = ops.list();
+
+ for (String table : allTables) {
+ SortedSet<Text> splits1 = new TreeSet<>();
+ splits1.add(new Text("split1"));
+ SortedSet<Text> splits2 = new TreeSet<>();
+ splits2.add(new Text("split2"));
+ SortedMap<Text,TabletMergeability> splits2Map =
+ TabletMergeabilityUtil.userDefaultSplits(splits2);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits1));
+ assertThrows(AccumuloException.class, () -> ops.putSplits(table, splits2Map));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits1);
+ ops.putSplits(table, splits2Map);
+ var listSplits = ops.listSplits(table);
+ assertTrue(listSplits.containsAll(splits1));
+ assertTrue(listSplits.containsAll(splits2));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class, () -> ops.rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
Review Comment:
comparing counts could cause failures because these tables are configured to compact aggresively. So if the table compacts after the flush and before counting is done in the test it could throw things off. Instead of comparing counts, could compare that the set of files changed.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,937 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TabletMergeability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.clientImpl.TabletMergeabilityUtil;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ userTable = null;
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps =
+ Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName);
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var untestedOps =
+ allTableOps.filter(op -> testMethodNames.stream().noneMatch(test -> test.contains(op)))
+ .collect(Collectors.toSet());
+ assertTrue(untestedOps.isEmpty(), "The table operations " + untestedOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ for (Map.Entry<String,String> entry : ops.tableIdMap().entrySet()) {
+ var tableName = entry.getKey();
+ var tableId = TableId.of(entry.getValue());
+ if (tableName.equals(AccumuloTable.ROOT.tableName())) {
+ assertEquals(AccumuloTable.ROOT.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.METADATA.tableName())) {
+ assertEquals(AccumuloTable.METADATA.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.FATE.tableName())) {
+ assertEquals(AccumuloTable.FATE.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.SCAN_REF.tableName())) {
+ assertEquals(AccumuloTable.SCAN_REF.tableId(), tableId);
+ } else if (tableName.equals(userTable)) {
+ assertFalse(AccumuloTable.allTableIds().contains(tableId));
+ } else {
+ throw new IllegalStateException("Unrecognized table: " + tableName);
+ }
+ }
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // addSplits,listSplits,putSplits tested elsewhere for METADATA, ROOT, and user tables, but
+ // testing here as well since this setup is needed to test for splitRangeByTablets anyway,
+ // which is untested elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = ops.list();
+
+ for (String table : allTables) {
+ SortedSet<Text> splits1 = new TreeSet<>();
+ splits1.add(new Text("split1"));
+ SortedSet<Text> splits2 = new TreeSet<>();
+ splits2.add(new Text("split2"));
+ SortedMap<Text,TabletMergeability> splits2Map =
+ TabletMergeabilityUtil.userDefaultSplits(splits2);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits1));
+ assertThrows(AccumuloException.class, () -> ops.putSplits(table, splits2Map));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits1);
+ ops.putSplits(table, splits2Map);
+ var listSplits = ops.listSplits(table);
+ assertTrue(listSplits.containsAll(splits1));
+ assertTrue(listSplits.containsAll(splits2));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class, () -> ops.rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
Review Comment:
Looking into this test noticed #5494
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,937 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TabletMergeability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.clientImpl.TabletMergeabilityUtil;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ userTable = null;
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps =
+ Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName);
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var untestedOps =
+ allTableOps.filter(op -> testMethodNames.stream().noneMatch(test -> test.contains(op)))
+ .collect(Collectors.toSet());
+ assertTrue(untestedOps.isEmpty(), "The table operations " + untestedOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ for (Map.Entry<String,String> entry : ops.tableIdMap().entrySet()) {
+ var tableName = entry.getKey();
+ var tableId = TableId.of(entry.getValue());
+ if (tableName.equals(AccumuloTable.ROOT.tableName())) {
+ assertEquals(AccumuloTable.ROOT.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.METADATA.tableName())) {
+ assertEquals(AccumuloTable.METADATA.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.FATE.tableName())) {
+ assertEquals(AccumuloTable.FATE.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.SCAN_REF.tableName())) {
+ assertEquals(AccumuloTable.SCAN_REF.tableId(), tableId);
+ } else if (tableName.equals(userTable)) {
+ assertFalse(AccumuloTable.allTableIds().contains(tableId));
+ } else {
+ throw new IllegalStateException("Unrecognized table: " + tableName);
+ }
+ }
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // addSplits,listSplits,putSplits tested elsewhere for METADATA, ROOT, and user tables, but
+ // testing here as well since this setup is needed to test for splitRangeByTablets anyway,
+ // which is untested elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = ops.list();
+
+ for (String table : allTables) {
+ SortedSet<Text> splits1 = new TreeSet<>();
+ splits1.add(new Text("split1"));
+ SortedSet<Text> splits2 = new TreeSet<>();
+ splits2.add(new Text("split2"));
+ SortedMap<Text,TabletMergeability> splits2Map =
+ TabletMergeabilityUtil.userDefaultSplits(splits2);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits1));
+ assertThrows(AccumuloException.class, () -> ops.putSplits(table, splits2Map));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits1);
+ ops.putSplits(table, splits2Map);
+ var listSplits = ops.listSplits(table);
+ assertTrue(listSplits.containsAll(splits1));
+ assertTrue(listSplits.containsAll(splits2));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class, () -> ops.rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
Review Comment:
Before initially setting the prop could add a check to ensure its not already there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050990559
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
Review Comment:
Good point, added testing for values as well in 4eb99ebd482139c834dbad28948a1842ab88b7d5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060459933
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
Review Comment:
deleteRows on system tables is no longer possible. Test now expects deleteRows to throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060803913
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
Review Comment:
Opened https://github.com/apache/accumulo/pull/5510
A lot of the table operations don't have javadoc for what the thrown exceptions mean (including `setTabletAvailability` in this case), so maybe this can be it's own ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060498700
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,937 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TabletMergeability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.clientImpl.TabletMergeabilityUtil;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ userTable = null;
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps =
+ Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName);
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var untestedOps =
+ allTableOps.filter(op -> testMethodNames.stream().noneMatch(test -> test.contains(op)))
+ .collect(Collectors.toSet());
+ assertTrue(untestedOps.isEmpty(), "The table operations " + untestedOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ for (Map.Entry<String,String> entry : ops.tableIdMap().entrySet()) {
+ var tableName = entry.getKey();
+ var tableId = TableId.of(entry.getValue());
+ if (tableName.equals(AccumuloTable.ROOT.tableName())) {
+ assertEquals(AccumuloTable.ROOT.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.METADATA.tableName())) {
+ assertEquals(AccumuloTable.METADATA.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.FATE.tableName())) {
+ assertEquals(AccumuloTable.FATE.tableId(), tableId);
+ } else if (tableName.equals(AccumuloTable.SCAN_REF.tableName())) {
+ assertEquals(AccumuloTable.SCAN_REF.tableId(), tableId);
+ } else if (tableName.equals(userTable)) {
+ assertFalse(AccumuloTable.allTableIds().contains(tableId));
+ } else {
+ throw new IllegalStateException("Unrecognized table: " + tableName);
+ }
+ }
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // addSplits,listSplits,putSplits tested elsewhere for METADATA, ROOT, and user tables, but
+ // testing here as well since this setup is needed to test for splitRangeByTablets anyway,
+ // which is untested elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = ops.list();
+
+ for (String table : allTables) {
+ SortedSet<Text> splits1 = new TreeSet<>();
+ splits1.add(new Text("split1"));
+ SortedSet<Text> splits2 = new TreeSet<>();
+ splits2.add(new Text("split2"));
+ SortedMap<Text,TabletMergeability> splits2Map =
+ TabletMergeabilityUtil.userDefaultSplits(splits2);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits1));
+ assertThrows(AccumuloException.class, () -> ops.putSplits(table, splits2Map));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits1);
+ ops.putSplits(table, splits2Map);
+ var listSplits = ops.listSplits(table);
+ assertTrue(listSplits.containsAll(splits1));
+ assertTrue(listSplits.containsAll(splits2));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class, () -> ops.rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
Review Comment:
Good point. Addressed in c88263153459d77dad6ce76bb3c7ac4449bdbde1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060572359
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
Review Comment:
With BULK_IMPORT2, the table is validated:
`final var tableId = validateTableIdArgument(arguments.get(0), tableOp, NOT_ROOT_TABLE_ID);`
Allowing tableIds that aren't the ROOT table to pass this check.
Then, the table is further validated to check if the user has permission to BULK_IMPORT2 into the table:
`canBulkImport = security.canBulkImport(c, tableId, tableName, dir, null, namespaceId);`
I think the validation of the tableId should include all system tables. I don't think we ever would want a user to bulk import to a system table, regardless of permissions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2049331010
##########
core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java:
##########
@@ -660,6 +660,7 @@ private void requestTabletHosting(ClientContext context,
}
// System tables should always be hosted
+ // TODO should this return for all sys tables?
if (AccumuloTable.ROOT.tableId() == tableId || AccumuloTable.METADATA.tableId() == tableId) {
return;
}
Review Comment:
> Yes it should return for all tablets, we always expect the system tablets to be hosted.
I think this might not be fully in place. `TableOperationsImpl.setTabletAvailability` uses the `NOT_BUILTIN_TABLE` validator to throw an IllegalArgumentException if someone tries to
set the TabletAvailability on a table in the `accumulo` namespace using the client.
However, `TabletMetadata.getTabletAvailability` only overrides the value to `HOSTED` for the root and metadata tables. `TabletMetadata.getTabletAvailability` is used by `TabletGoalState.compute`, which is used by the TabletGroupWatcher. It might be possible to someone to do surgery on the `accumulo.metadata` table and manually set the `fate` and `scanref` tables to `UNHOSTED` or `ONDEMAND`.
I'll create a PR to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2042455425
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
Review Comment:
This will be changed once #5433 is completed and merged in: should not be able to offline any system table
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
Review Comment:
deleteRows() is allowed for FATE table but probably shouldn't be. Can someone confirm that this is an action that should not be allowed? Not testing for now since deleting our FATE operation from the FATE table when it is in progress will cause problems in the MAC and there is no way to put junk data in the FATE table with the Manager running.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
Review Comment:
Skipping ROOT for now until I resolve the bug in TableDiskUsage.getDiskUsage... Will always show a disk usage of 0 for the ROOT table as it is currently scanning the METADATA table for the ROOT table metadata.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
Review Comment:
Should maybe be an AccumuloException to keep consistency with the other table operations methods that throw an AccumuloException when performed on an unacceptable table?
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
Review Comment:
deleteRows() is allowed for SCAN_REF table but probably shouldn't be. Can someone confirm that this is an action that should not be allowed? Testing it here is fine for now since just some junk data in the SCAN_REF table anyways but should probably result in an Exception like METADATA and ROOT.
##########
core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java:
##########
@@ -660,6 +660,7 @@ private void requestTabletHosting(ClientContext context,
}
// System tables should always be hosted
+ // TODO should this return for all sys tables?
if (AccumuloTable.ROOT.tableId() == tableId || AccumuloTable.METADATA.tableId() == tableId) {
return;
}
Review Comment:
Commenting here as a reminder to delete this once resolved
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
Review Comment:
TLDR: Test does not work. Potential issue with compacting system tables but more likely issue with testing approach.
This test is not working as expected. Only passes when compacting the FATE table is done last. The root cause of the issue is the call to createFateTableRow(userTable). This initiates a very slow compaction on the userTable that will never complete until cancelled. This is done to create some data in the FATE table and flush the table before we compact. Compacting the FATE table with this compaction FATE op in the FATE table works as expected (compaction completes on FATE table as expected). However, compacting subsequent system tables do not work as expected. One way to see this is putting FATE before SCAN_REF in the list. You can see that the compaction on SCAN_REF hangs indefinitely. It gets stuck on the CompactionDriver.isReady() step. Considered that the slow compaction which would still be running on the userTable is somehow causing the issue, but even if the compaction is cancelled and we wait for it to be completely cancelled and cleaned up from the FATE table, the issue s
till occurs on subsequent system table compactions. Further, the compaction on the userTable should have no effect on compactions on the system tables since compacting the userTable won't lock the `accumulo` namespace or anything like that. I have not been able to figure out this issue, if someone can spot an issue with my testing approach or look at the Manager log during the described failure case and figure out what is wrong, that would be much appreciated.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
Review Comment:
Some throw AccumuloException and some throw AccumuloSecurityException not sure if that matters. The AccumuloException occurs from FateServiceHandler and the AccumuloSecurityException occurs from ClientServiceHandler. The validation done in FateServiceHandler only checks that it's not the ROOT table. Should probably check that it's not a built in table. This would result in the same Exception for all system tables and is more clear.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
Review Comment:
Should setting properties on the system tables continue to be allowed or is this undesired?
##########
test/src/main/java/org/apache/accumulo/test/functional/CloneTestIT.java:
##########
@@ -352,18 +352,18 @@ public void testCloneWithSplits() throws Exception {
}
@Test
- public void testCloneRootTable() {
+ public void testCloneSystemTables() {
try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- assertThrows(AccumuloException.class, () -> client.tableOperations()
- .clone(AccumuloTable.ROOT.tableName(), "rc1", CloneConfiguration.empty()));
- }
- }
-
- @Test
- public void testCloneMetadataTable() {
- try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- assertThrows(AccumuloException.class, () -> client.tableOperations()
- .clone(AccumuloTable.METADATA.tableName(), "mc1", CloneConfiguration.empty()));
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var cloneTableName = tableNames[i];
+ assertThrows(Exception.class, () -> client.tableOperations().clone(sysTable.tableName(),
+ cloneTableName, CloneConfiguration.empty()));
Review Comment:
Same comment about some throwing AccumuloException and some throwing AccumuloSecurityException
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
Review Comment:
This will be changed once #5433 is completed and merged in: should not be able to offline any system table
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
Review Comment:
Should adding constraints on the system tables continue to be allowed or is this undesired?
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
Review Comment:
Should custom iterators on the system tables continue to be allowed or is this undesired?
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
+ assertTrue(ops.getTabletInformation(sysTable.tableName(), new Range()).findAny().isPresent());
+ ops.getTabletInformation(sysTable.tableName(), new Range())
+ .forEach(ti -> assertEquals(TabletAvailability.HOSTED, ti.getTabletAvailability()));
+ }
+ }
+
+ /**
+ * Creates some data in the FATE table. Will create a single row each call. Does so by initiating
+ * a very slow compaction on the given table.
+ *
+ * @param table the table the fate operation will operate on
+ * @return the row created
+ */
+ private Text createFateTableRow(String table) throws Exception {
+ attachSlowMajcIterator(table);
+ ReadWriteIT.ingest(client, 5, 5, 5, 0, table);
+
+ Set<Text> rowsSeenBeforeNewOp = new HashSet<>();
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ for (var entry : scanner) {
+ rowsSeenBeforeNewOp.add(entry.getKey().getRow());
+ }
+ }
+
+ // start a very slow compaction to create a FATE op that will linger in the FATE table until
+ // cancelled
+ client.tableOperations().compact(table, null, null, true, false);
+
+ Set<Text> rowsSeenAfterNewOp = new HashSet<>();
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ for (var entry : scanner) {
+ rowsSeenAfterNewOp.add(entry.getKey().getRow());
+ }
+ }
+
+ var newOp = Sets.difference(rowsSeenAfterNewOp, rowsSeenBeforeNewOp);
+ assertEquals(1, newOp.size());
+ return newOp.stream().findFirst().orElseThrow();
+ }
+
+ private void attachSlowMajcIterator(String table) throws Exception {
+ if (!ops.listIterators(table).containsKey(SLOW_ITER_NAME)) {
+ IteratorSetting is = new IteratorSetting(1, SLOW_ITER_NAME, SlowIterator.class);
+ is.addOption("sleepTime", "60000");
+ ops.attachIterator(table, is, EnumSet.of(IteratorUtil.IteratorScope.majc));
+ }
+ }
+
+ private void removeSlowMajcIterator(String table) throws Exception {
+ ops.removeIterator(table, SLOW_ITER_NAME, EnumSet.of(IteratorUtil.IteratorScope.majc));
+ }
+
+ /**
+ * Creates some data in the SCAN_REF table. Will create a single row each call.
+ *
+ * @return the row created
+ */
+ private Text createScanRefTableRow() {
+ HostAndPort server = HostAndPort.fromParts("127.0.0.1", 1234);
+ UUID serverLockUUID = UUID.randomUUID();
+ Set<ScanServerRefTabletFile> scanRefs = Stream.of("F0000070.rf", "F0000071.rf")
+ .map(f -> "hdfs://localhost:8020/accumulo/tables/2a/default_tablet/" + f)
+ .map(f -> new ScanServerRefTabletFile(f, server.toString(), serverLockUUID))
+ .collect(Collectors.toSet());
+ getCluster().getServerContext().getAmple().scanServerRefs().put(scanRefs);
+ return new Text(getCluster().getServerContext().getAmple().scanServerRefs().list()
+ .filter(tf -> tf.getServerLockUUID().equals(serverLockUUID)).findFirst().orElseThrow()
+ .getServerLockUUID().toString());
+ }
+
+ /**
+ * Cleans up the data in the FATE table that was created by calls to
+ * {@link #createFateTableRow(String)}
+ */
+ private void cleanupFateTable() throws Exception {
+ ops.cancelCompaction(userTable);
+ // Wait for FATE table to be clear
+ Wait.waitFor(() -> {
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ return !scanner.iterator().hasNext();
+ }
+ });
+ }
+
+ /**
+ * Cleans up the data in the SCAN_REF table that was created by calls to
+ * {@link #createScanRefTableRow()}
+ */
+ private void cleanupScanRefTable() throws Exception {
+ ops.deleteRows(AccumuloTable.SCAN_REF.tableName(), null, null);
Review Comment:
Same comment about deleteRows() here, but using it for now since it's helpful cleanup in this test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2052470178
##########
core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java:
##########
@@ -660,6 +660,7 @@ private void requestTabletHosting(ClientContext context,
}
// System tables should always be hosted
+ // TODO should this return for all sys tables?
if (AccumuloTable.ROOT.tableId() == tableId || AccumuloTable.METADATA.tableId() == tableId) {
return;
}
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2051079475
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
Review Comment:
Yeah, it will pass on its own but not when all tests are run because some other tests call `createFateTableRow(userTable)` which is the root cause of the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054319430
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
Review Comment:
I can take a look into this. If this is not figured out before this PR is merged then we should open an issue for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054328436
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
Review Comment:
This unreleased code, so now is the time to change exceptions. Makes sense to me to change it to AccumuloException. One important thing is making sure the expected exceptions are noted in the javadoc, that helps developers and users know what is expected going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 merged PR #5474:
URL: https://github.com/apache/accumulo/pull/5474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2167239982
##########
test/src/main/java/org/apache/accumulo/test/functional/CloneTestIT.java:
##########
@@ -352,18 +352,18 @@ public void testCloneWithSplits() throws Exception {
}
@Test
- public void testCloneRootTable() {
+ public void testCloneSystemTables() {
try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- assertThrows(AccumuloException.class, () -> client.tableOperations()
- .clone(AccumuloTable.ROOT.tableName(), "rc1", CloneConfiguration.empty()));
- }
- }
-
- @Test
- public void testCloneMetadataTable() {
- try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- assertThrows(AccumuloException.class, () -> client.tableOperations()
- .clone(AccumuloTable.METADATA.tableName(), "mc1", CloneConfiguration.empty()));
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var cloneTableName = tableNames[i];
+ assertThrows(Exception.class, () -> client.tableOperations().clone(sysTable.tableName(),
+ cloneTableName, CloneConfiguration.empty()));
Review Comment:
Created #5680
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
Review Comment:
Created #5680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050810191
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
Review Comment:
Not sure if this is an improvement but tried this out while looking at the code:
```
var untestedOps = allTableOps.stream()
.filter(op -> testMethodNames.stream().noneMatch(test -> test.contains(op)))
.collect(Collectors.toSet());
```
This could replace the iterator and while loop parts.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
Review Comment:
When I run this test case on its own it passes. But does time out when I run all tests in this class.
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
Review Comment:
```suggestion
if (userTable != null && ops.exists(userTable)) {
cleanupFateTable();
ops.delete(userTable);
userTable = null;
}
```
Could make `userTable` null to avoid the exists check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2050916597
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,958 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
Review Comment:
```suggestion
if (userTable != null && ops.exists(userTable)) {
cleanupFateTable();
ops.delete(userTable);
userTable = null;
}
```
Could make `userTable` null after its deleted to avoid the exists check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2052578751
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
+ assertTrue(ops.getTabletInformation(sysTable.tableName(), new Range()).findAny().isPresent());
+ ops.getTabletInformation(sysTable.tableName(), new Range())
+ .forEach(ti -> assertEquals(TabletAvailability.HOSTED, ti.getTabletAvailability()));
+ }
+ }
+
+ /**
+ * Creates some data in the FATE table. Will create a single row each call. Does so by initiating
+ * a very slow compaction on the given table.
+ *
+ * @param table the table the fate operation will operate on
+ * @return the row created
+ */
+ private Text createFateTableRow(String table) throws Exception {
+ attachSlowMajcIterator(table);
+ ReadWriteIT.ingest(client, 5, 5, 5, 0, table);
+
+ Set<Text> rowsSeenBeforeNewOp = new HashSet<>();
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ for (var entry : scanner) {
+ rowsSeenBeforeNewOp.add(entry.getKey().getRow());
+ }
+ }
+
+ // start a very slow compaction to create a FATE op that will linger in the FATE table until
+ // cancelled
+ client.tableOperations().compact(table, null, null, true, false);
+
+ Set<Text> rowsSeenAfterNewOp = new HashSet<>();
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ for (var entry : scanner) {
+ rowsSeenAfterNewOp.add(entry.getKey().getRow());
+ }
+ }
+
+ var newOp = Sets.difference(rowsSeenAfterNewOp, rowsSeenBeforeNewOp);
+ assertEquals(1, newOp.size());
+ return newOp.stream().findFirst().orElseThrow();
+ }
+
+ private void attachSlowMajcIterator(String table) throws Exception {
+ if (!ops.listIterators(table).containsKey(SLOW_ITER_NAME)) {
+ IteratorSetting is = new IteratorSetting(1, SLOW_ITER_NAME, SlowIterator.class);
+ is.addOption("sleepTime", "60000");
+ ops.attachIterator(table, is, EnumSet.of(IteratorUtil.IteratorScope.majc));
+ }
+ }
+
+ private void removeSlowMajcIterator(String table) throws Exception {
+ ops.removeIterator(table, SLOW_ITER_NAME, EnumSet.of(IteratorUtil.IteratorScope.majc));
+ }
+
+ /**
+ * Creates some data in the SCAN_REF table. Will create a single row each call.
+ *
+ * @return the row created
+ */
+ private Text createScanRefTableRow() {
+ HostAndPort server = HostAndPort.fromParts("127.0.0.1", 1234);
+ UUID serverLockUUID = UUID.randomUUID();
+ Set<ScanServerRefTabletFile> scanRefs = Stream.of("F0000070.rf", "F0000071.rf")
+ .map(f -> "hdfs://localhost:8020/accumulo/tables/2a/default_tablet/" + f)
+ .map(f -> new ScanServerRefTabletFile(f, server.toString(), serverLockUUID))
+ .collect(Collectors.toSet());
+ getCluster().getServerContext().getAmple().scanServerRefs().put(scanRefs);
+ return new Text(getCluster().getServerContext().getAmple().scanServerRefs().list()
+ .filter(tf -> tf.getServerLockUUID().equals(serverLockUUID)).findFirst().orElseThrow()
+ .getServerLockUUID().toString());
+ }
+
+ /**
+ * Cleans up the data in the FATE table that was created by calls to
+ * {@link #createFateTableRow(String)}
+ */
+ private void cleanupFateTable() throws Exception {
+ ops.cancelCompaction(userTable);
+ // Wait for FATE table to be clear
+ Wait.waitFor(() -> {
+ try (var scanner = client.createScanner(AccumuloTable.FATE.tableName())) {
+ return !scanner.iterator().hasNext();
+ }
+ });
+ }
+
+ /**
+ * Cleans up the data in the SCAN_REF table that was created by calls to
+ * {@link #createScanRefTableRow()}
+ */
+ private void cleanupScanRefTable() throws Exception {
+ ops.deleteRows(AccumuloTable.SCAN_REF.tableName(), null, null);
Review Comment:
Created https://github.com/apache/accumulo/pull/5491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2052578170
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
Review Comment:
Created https://github.com/apache/accumulo/pull/5491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2054309673
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
Review Comment:
That could be dangerous, but I think its fine to leave it because there are many other dangerous things that could potentially be done. Overall the best option is to ensure the metadata table has locked down permissions for setting table properties and writing to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2060803913
##########
test/src/main/java/org/apache/accumulo/test/ComprehensiveTableOperationsIT.java:
##########
@@ -0,0 +1,957 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Accumulo;
+import org.apache.accumulo.core.client.AccumuloClient;
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.IteratorSetting;
+import org.apache.accumulo.core.client.admin.TableOperations;
+import org.apache.accumulo.core.client.admin.TabletAvailability;
+import org.apache.accumulo.core.client.admin.TimeType;
+import org.apache.accumulo.core.client.summary.SummarizerConfiguration;
+import org.apache.accumulo.core.clientImpl.ClientContext;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.Mutation;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.Filter;
+import org.apache.accumulo.core.iterators.IteratorUtil;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.metadata.AccumuloTable;
+import org.apache.accumulo.core.metadata.ScanServerRefTabletFile;
+import org.apache.accumulo.core.metadata.StoredTabletFile;
+import org.apache.accumulo.core.metadata.schema.TabletsMetadata;
+import org.apache.accumulo.core.security.Authorizations;
+import org.apache.accumulo.harness.SharedMiniClusterBase;
+import org.apache.accumulo.minicluster.ServerType;
+import org.apache.accumulo.test.functional.BasicSummarizer;
+import org.apache.accumulo.test.functional.BulkNewIT;
+import org.apache.accumulo.test.functional.CloneTestIT;
+import org.apache.accumulo.test.functional.CompactionIT;
+import org.apache.accumulo.test.functional.ConstraintIT;
+import org.apache.accumulo.test.functional.DeleteRowsIT;
+import org.apache.accumulo.test.functional.LocalityGroupIT;
+import org.apache.accumulo.test.functional.ManagerAssignmentIT;
+import org.apache.accumulo.test.functional.MergeTabletsIT;
+import org.apache.accumulo.test.functional.ReadWriteIT;
+import org.apache.accumulo.test.functional.RenameIT;
+import org.apache.accumulo.test.functional.SlowIterator;
+import org.apache.accumulo.test.functional.SummaryIT;
+import org.apache.accumulo.test.functional.TabletAvailabilityIT;
+import org.apache.accumulo.test.util.Wait;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Sets;
+import com.google.common.net.HostAndPort;
+
+/**
+ * A comprehensive IT of all table operations against user tables and all system tables while
+ * avoiding duplicating existing testing. This does not test for edge cases, but rather tests for
+ * basic expected functionality of all table operations against user tables and all system tables.
+ */
+public class ComprehensiveTableOperationsIT extends SharedMiniClusterBase {
+ private static final Logger log = LoggerFactory.getLogger(ComprehensiveTableOperationsIT.class);
+ private static final String SLOW_ITER_NAME = "CustomSlowIter";
+ private AccumuloClient client;
+ private TableOperations ops;
+ private String userTable;
+
+ @Override
+ protected Duration defaultTimeout() {
+ return Duration.ofMinutes(1);
+ }
+
+ @BeforeAll
+ public static void setup() throws Exception {
+ SharedMiniClusterBase.startMiniCluster();
+ }
+
+ @AfterAll
+ public static void teardown() {
+ SharedMiniClusterBase.stopMiniCluster();
+ }
+
+ @BeforeEach
+ public void beforeEach() {
+ client = Accumulo.newClient().from(getClientProps()).build();
+ ops = client.tableOperations();
+ }
+
+ @AfterEach
+ public void afterEach() throws Exception {
+ // ensure none of the FATE or SCAN_REF data we created persists between tests. Also ensure the
+ // user table does not persist between tests
+ if (userTable != null && ops.exists(userTable)) {
+ cleanupFateTable();
+ ops.delete(userTable);
+ }
+ cleanupScanRefTable();
+ client.close();
+ }
+
+ @Test
+ public void testAllTested() {
+ var allTableOps = Arrays.stream(TableOperations.class.getDeclaredMethods()).map(Method::getName)
+ .collect(Collectors.toSet());
+ var testMethodNames = Arrays.stream(ComprehensiveTableOperationsIT.class.getDeclaredMethods())
+ .map(Method::getName).collect(Collectors.toSet());
+ var allTableOpsIter = allTableOps.iterator();
+ while (allTableOpsIter.hasNext()) {
+ var tableOp = allTableOpsIter.next();
+ for (var testMethodName : testMethodNames) {
+ if (testMethodName.contains(tableOp)) {
+ allTableOpsIter.remove();
+ break;
+ }
+ }
+ }
+ assertTrue(allTableOps.isEmpty(), "The table operations " + allTableOps + " are untested");
+ }
+
+ @Test
+ public void testExpectedSystemTables() {
+ var testedSystemTableIds =
+ Set.of(AccumuloTable.ROOT.tableId(), AccumuloTable.METADATA.tableId(),
+ AccumuloTable.FATE.tableId(), AccumuloTable.SCAN_REF.tableId());
+ assertEquals(testedSystemTableIds, AccumuloTable.allTableIds(),
+ "There are newly added system tables that are untested in this IT. Ensure each "
+ + "test has testing for the new system table");
+ }
+
+ @Test
+ public void test_list_tableIdMap() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables are returned
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ assertEquals(expected, ops.list());
+ assertEquals(expected, ops.tableIdMap().keySet());
+ }
+
+ @Test
+ public void test_exists() throws Exception {
+ // Thoroughly tested elsewhere, but simple enough to test here. Test that all system tables
+ // and user tables exist
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var expected = Arrays.stream(AccumuloTable.values()).map(AccumuloTable::tableName)
+ .collect(Collectors.toSet());
+ expected.add(userTable);
+ for (String table : expected) {
+ assertTrue(ops.exists(table));
+ }
+ }
+
+ @Test
+ public void test_create() throws Exception {
+ // Creating user tables is thoroughly tested. Make sure we can't create any of the already
+ // existing system tables, though.
+ for (var systemTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.create(systemTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_exportTable_importTable() throws Exception {
+ // exportTable, importTable for user tables is tested in ImportExportIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ImportExportIT.class.getName()));
+ // exportTable, importTable untested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ Path baseDir = ImportExportIT.createBaseDir(getCluster(), getClass());
+ var fs = getCluster().getFileSystem();
+
+ // export the user table, so we can test importing into the system tables
+ ops.offline(userTable, true);
+ Path exportUserDir;
+ exportUserDir = new Path(baseDir, "export_userdir");
+ fs.deleteOnExit(exportUserDir);
+ ops.exportTable(userTable, exportUserDir.toString());
+
+ var sysTables = AccumuloTable.values();
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ Path exportDir = new Path(baseDir, "export" + i);
+ fs.deleteOnExit(exportDir);
+
+ // not offline, can't export
+ assertThrows(IllegalStateException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ // can't offline, so will never be able to export
+ assertThrows(AccumuloException.class, () -> ops.offline(sysTable.tableName(), true));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ break;
+ case FATE:
+ case SCAN_REF:
+ ops.offline(sysTable.tableName(), true);
+ try {
+ assertThrows(AccumuloException.class,
+ () -> ops.exportTable(sysTable.tableName(), exportDir.toString()));
+ assertThrows(AccumuloException.class,
+ () -> ops.importTable(sysTable.tableName(), exportUserDir.toString()));
+ } finally {
+ ops.online(sysTable.tableName(), true);
+ }
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_addSplits_putSplits_listSplits_splitRangeByTablets() throws Exception {
+ // note that addSplits and putSplits are implemented the same, just take different args. No
+ // need to test both
+
+ // addSplits, listSplits tested elsewhere for METADATA, ROOT, and user tables, but testing here
+ // as well since this setup is needed to test for splitRangeByTablets anyway, which is untested
+ // elsewhere
+
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ var allTables = new ArrayList<>(ops.list());
+
+ for (int i = 0; i < allTables.size(); i++) {
+ SortedSet<Text> splits = new TreeSet<>();
+ splits.add(new Text(i + ""));
+ String table = allTables.get(i);
+
+ if (table.equals(AccumuloTable.ROOT.tableName())) {
+ // cannot add splits to ROOT
+ assertThrows(AccumuloException.class, () -> ops.addSplits(table, splits));
+ assertEquals(0, ops.listSplits(table).size());
+ } else {
+ ops.addSplits(table, splits);
+ assertTrue(ops.listSplits(table).containsAll(splits));
+ }
+
+ assertEquals(ops.splitRangeByTablets(table, new Range(), 99).size(),
+ ops.listSplits(table).size() + 1);
+ }
+ }
+
+ @Test
+ public void test_locate() throws Exception {
+ // locate for user tables is tested in LocatorIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(LocatorIT.class.getName()));
+ // locate for METADATA and ROOT tables is tested in ManagerAssignmentIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ManagerAssignmentIT.class.getName()));
+
+ // basic functionality check for locate on FATE and SCAN_REF tables
+ var fateLocations =
+ ops.locate(AccumuloTable.FATE.tableName(), Collections.singletonList(new Range()));
+ var fateGroupByTablet = fateLocations.groupByTablet().keySet();
+ assertFalse(fateGroupByTablet.isEmpty());
+ fateGroupByTablet.forEach(tid -> {
+ var tabletLoc = fateLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ var scanRefLocations =
+ ops.locate(AccumuloTable.SCAN_REF.tableName(), Collections.singletonList(new Range()));
+ var scanRefGroupByTablet = scanRefLocations.groupByTablet().keySet();
+ assertFalse(scanRefGroupByTablet.isEmpty());
+ scanRefGroupByTablet.forEach(tid -> {
+ var tabletLoc = scanRefLocations.getTabletLocation(tid);
+ assertNotNull(tabletLoc);
+ assertTrue(tabletLoc.contains(":"));
+ });
+ }
+
+ @Test
+ public void test_getMaxRow_deleteRows() throws Exception {
+ // getMaxRow for user tables is tested in FindMaxIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(FindMaxIT.class.getName()));
+ // getMaxRow not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ var maxRow =
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true);
+ log.info("Max row of {} : {}", sysTable, maxRow);
+ assertNotNull(maxRow);
+ }
+
+ // deleteRows for user tables is tested in DeleteRowsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(DeleteRowsIT.class.getName()));
+ // deleteRows not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case METADATA:
+ case ROOT:
+ // should not be able to delete rows of METADATA or ROOT tables
+ assertThrows(AccumuloException.class,
+ () -> ops.deleteRows(sysTable.tableName(), null, null));
+ break;
+ case FATE:
+ break;
+ case SCAN_REF:
+ cleanupScanRefTable();
+ // check max row again
+ assertNull(
+ ops.getMaxRow(sysTable.tableName(), Authorizations.EMPTY, null, true, null, true));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_merge() throws Exception {
+ // merge for user tables is tested in various ITs. One example is MergeTabletsIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(MergeTabletsIT.class.getName()));
+ // merge for METADATA and ROOT system tables tested in MetaSplitIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(MetaSplitIT.class.getName()));
+ // merge for FATE and SCAN_REF tables not tested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ var fateRow1 = createFateTableRow(userTable);
+ createFateTableRow(userTable); // fate row 2
+ var scanRefRow1 = createScanRefTableRow();
+ createScanRefTableRow(); // scan ref row 2
+
+ SortedSet<Text> fateSplits = new TreeSet<>();
+ fateSplits.add(new Text(fateRow1));
+ ops.addSplits(AccumuloTable.FATE.tableName(), fateSplits);
+ // there may be preexisting splits on the FATE table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).contains(fateRow1));
+ ops.merge(AccumuloTable.FATE.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.FATE.tableName()).isEmpty());
+
+ SortedSet<Text> scanRefSplits = new TreeSet<>();
+ scanRefSplits.add(new Text(scanRefRow1));
+ ops.addSplits(AccumuloTable.SCAN_REF.tableName(), scanRefSplits);
+ // there may be preexisting splits on the SCAN_REF table from a previous test
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).contains(scanRefRow1));
+ ops.merge(AccumuloTable.SCAN_REF.tableName(), null, null);
+ assertTrue(ops.listSplits(AccumuloTable.SCAN_REF.tableName()).isEmpty());
+ }
+
+ @Test
+ public void test_compact() throws Exception {
+ // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // disable the GC to prevent automatic compactions on METADATA and ROOT tables
+ getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
+ try {
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // create some RFiles for the METADATA and ROOT tables by creating some data in the user
+ // table, flushing that table, then the METADATA table, then the ROOT table
+ for (int i = 0; i < 3; i++) {
+ try (var bw = client.createBatchWriter(userTable)) {
+ var mut = new Mutation("r" + i);
+ mut.put("cf", "cq", "v");
+ bw.addMutation(mut);
+ }
+ ops.flush(userTable, null, null, true);
+ ops.flush(AccumuloTable.METADATA.tableName(), null, null, true);
+ ops.flush(AccumuloTable.ROOT.tableName(), null, null, true);
+ }
+
+ for (var sysTable : List.of(AccumuloTable.ROOT, AccumuloTable.METADATA,
+ AccumuloTable.SCAN_REF, AccumuloTable.FATE)) {
+ // create some RFiles for FATE and SCAN_REF tables
+ if (sysTable == AccumuloTable.SCAN_REF) {
+ createScanRefTableRow();
+ ops.flush(AccumuloTable.SCAN_REF.tableName(), null, null, true);
+ } else if (sysTable == AccumuloTable.FATE) {
+ createFateTableRow(userTable);
+ ops.flush(AccumuloTable.FATE.tableName(), null, null, true);
+ }
+
+ Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, true);
+ log.info("Finished compacting " + sysTable);
+
+ // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
+ // with 'A' that was not present before the compaction.
+ Wait.waitFor(() -> {
+ var stfsAfterCompact = getStoredTabFiles(sysTable);
+ String regex = "^A.*\\.rf$";
+ var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ var A_stfsAfterCompaction = stfsAfterCompact.stream()
+ .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
+ return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
+ });
+ }
+ } finally {
+ getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
+ }
+ }
+
+ @Test
+ public void test_cancelCompaction() throws Exception {
+ // cancelCompaction for user tables is tested in various ITs. One example is TableOperationsIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // test basic functionality for system tables
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ // Need some data in all the system tables. This allows the slow iterator we attach to the
+ // system table to work when we compact. ROOT and METADATA will already have data, so just need
+ // to create some data for the other system tables
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : AccumuloTable.values()) {
+ try {
+ var zrw = getCluster().getServerContext().getZooSession().asReaderWriter();
+ attachSlowMajcIterator(sysTable.tableName());
+
+ var metaFatesBeforeCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+
+ log.info("Compacting " + sysTable);
+ ops.compact(sysTable.tableName(), null, null, true, false);
+ log.info("Initiated compaction for " + sysTable);
+
+ // Wait for the compaction to be started
+ Wait.waitFor(() -> {
+ var metaFatesAfterCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return !Sets.difference(metaFatesAfterCompact, metaFatesBeforeCompact).isEmpty();
+ });
+
+ log.info("Cancelling compaction " + sysTable);
+ ops.cancelCompaction(sysTable.tableName());
+
+ // We can be sure that the compaction has been cancelled once we see no FATE operations
+ Wait.waitFor(() -> {
+ Set<String> metaFatesAfterCancelCompact = new HashSet<>(zrw.getChildren(Constants.ZFATE));
+ return metaFatesAfterCancelCompact.isEmpty();
+ });
+ } finally {
+ removeSlowMajcIterator(sysTable.tableName());
+ }
+ }
+ }
+
+ @Test
+ public void test_delete() throws Exception {
+ // delete for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // delete not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(AccumuloException.class, () -> ops.delete(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_clone() throws Exception {
+ // cloning user and system tables is tested in CloneTestIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(CloneTestIT.class.getName()));
+ }
+
+ @Test
+ public void test_rename() throws Exception {
+ // rename for user tables is tested in RenameIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(RenameIT.class.getName()));
+ // rename not tested for system tables. Test basic functionality here
+ var sysTables = AccumuloTable.values();
+ var tableNames = getUniqueNames(sysTables.length);
+
+ for (int i = 0; i < sysTables.length; i++) {
+ var sysTable = sysTables[i];
+ var tableName = tableNames[i];
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().rename(sysTable.tableName(), tableName));
+ }
+ }
+
+ @Test
+ public void test_flush() throws Exception {
+ // flush for user tables and the METADATA and ROOT tables is tested in CompactionIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
+ // flush for FATE and SCAN_REF not tested for. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+
+ for (var sysTable : Set.of(AccumuloTable.FATE, AccumuloTable.SCAN_REF)) {
+ int numFilesBeforeFlush = 0;
+
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ for (var tm : tabletsMetadata) {
+ numFilesBeforeFlush += tm.getFiles().size();
+ }
+ }
+ ops.flush(sysTable.tableName(), null, null, true);
+ // Wait until we see the new file
+ final int finalNumFilesBeforeFlush = numFilesBeforeFlush;
+ Wait.waitFor(() -> {
+ try (TabletsMetadata tabletsMetadata = getCluster().getServerContext().getAmple()
+ .readTablets().forTable(sysTable.tableId()).build()) {
+ int numFilesAfterFlush = 0;
+ for (var tm : tabletsMetadata) {
+ numFilesAfterFlush += tm.getFiles().size();
+ }
+ return numFilesAfterFlush > finalNumFilesBeforeFlush;
+ }
+ });
+ }
+ }
+
+ @Test
+ public void
+ test_setProperty_modifyProperties_removeProperty_getProperties_getTableProperties_getConfiguration()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (String tableName : allTables) {
+ // setProperty
+ String propKey = Property.TABLE_ARBITRARY_PROP_PREFIX.getKey() + "prop";
+ String propVal = "val";
+ String newPropVal = "newval";
+ ops.setProperty(tableName, propKey, propVal);
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, propVal));
+ assertEquals(propVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(propVal, ops.getConfiguration(tableName).get(propKey));
+ // modifyProperties
+ ops.modifyProperties(tableName, properties -> properties.put(propKey, newPropVal));
+ // getProperties, getTableProperties, and getConfiguration
+ assertTrue(propFound(tableName, propKey, newPropVal));
+ assertEquals(newPropVal, ops.getTableProperties(tableName).get(propKey));
+ assertEquals(newPropVal, ops.getConfiguration(tableName).get(propKey));
+ // removeProperty
+ ops.removeProperty(tableName, propKey);
+ // getProperties, getTableProperties, and getConfiguration
+ assertFalse(propFound(tableName, propKey, newPropVal));
+ assertNull(ops.getTableProperties(tableName).get(propKey));
+ assertNull(ops.getConfiguration(tableName).get(propKey));
+ }
+ }
+
+ @Test
+ public void test_setLocalityGroups_getLocalityGroups() throws Exception {
+ // setLocalityGroups, getLocalityGroups for user tables is tested in LocalityGroupIT. Ensure
+ // test exists
+ assertDoesNotThrow(() -> Class.forName(LocalityGroupIT.class.getName()));
+ // setLocalityGroups, getLocalityGroups for system tables not tested for. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ LocalityGroupIT.createAndSetLocalityGroups(client, sysTable.tableName());
+ LocalityGroupIT.verifyLocalityGroupSet(client, sysTable.tableName());
+ }
+ }
+
+ @Test
+ public void test_importDirectory() throws Exception {
+ // importDirectory for user tables is tested in BulkNewIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(BulkNewIT.class.getName()));
+ // importDirectory for system tables not tested for. Test basic functionality here
+ var rootPath = getCluster().getTemporaryPath().toString();
+ var dir = rootPath + "/" + getUniqueNames(1)[0];
+ BulkNewIT.writeData(dir + "/f1.", getCluster().getServerContext().getConfiguration(), 0, 5, 0);
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertThrows(Exception.class, () -> ops.importDirectory(dir).to(sysTable.tableName()).load());
+ }
+ }
+
+ @Test
+ public void test_offline_online_isOnline() throws Exception {
+ // offline,online,isOnline for user tables is tested in ComprehensiveBaseIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ComprehensiveBaseIT.class.getName()));
+ // offline,online,isOnline not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ switch (sysTable) {
+ case ROOT:
+ case METADATA:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ assertThrows(AccumuloException.class,
+ () -> client.tableOperations().offline(sysTable.tableName(), true));
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ break;
+ case SCAN_REF:
+ case FATE:
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().offline(sysTable.tableName(), true);
+ assertFalse(client.tableOperations().isOnline(sysTable.tableName()));
+ client.tableOperations().online(sysTable.tableName(), true);
+ assertTrue(client.tableOperations().isOnline(sysTable.tableName()));
+ break;
+ default:
+ throw new IllegalStateException("Unrecognized table: " + sysTable);
+ }
+ }
+ }
+
+ @Test
+ public void test_clearLocatorCache() throws Exception {
+ // clearLocatorCache is tested elsewhere for user tables but not for system tables. Simple
+ // enough to test all tables here
+ var clientContext = (ClientContext) client;
+ userTable = getUniqueNames(1)[0];
+
+ client.tableOperations().create(userTable);
+ ReadWriteIT.ingest(clientContext, 5, 5, 5, 0, userTable);
+ assertTrue(clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount() > 0);
+ ops.clearLocatorCache(userTable);
+ assertEquals(0,
+ clientContext.getTabletLocationCache(TableId.of(ops.tableIdMap().get(userTable)))
+ .getTabletHostingRequestCount());
+
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ ops.clearLocatorCache(sysTable.tableName());
+ assertEquals(0,
+ clientContext.getTabletLocationCache(sysTable.tableId()).getTabletHostingRequestCount());
+ }
+ }
+
+ @Test
+ public void
+ test_attachIterator_removeIterator_getIteratorSetting_listIterators_checkIteratorConflicts()
+ throws Exception {
+ // These may be tested elsewhere across several tests, but simpler to test all here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ String iterName = "iter_name";
+ var iterSetting = new IteratorSetting(100, iterName, NoOpFilter.class);
+ var scope = EnumSet.of(IteratorUtil.IteratorScope.majc, IteratorUtil.IteratorScope.minc);
+ // system and user tables
+ Set<String> allTables = ops.list();
+
+ for (String table : allTables) {
+ ops.attachIterator(table, iterSetting, scope);
+ try {
+ assertTrue(ops.listIterators(table).containsKey(iterName));
+ assertNull(ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.scan));
+ assertEquals(iterSetting,
+ ops.getIteratorSetting(table, iterName, IteratorUtil.IteratorScope.majc));
+ assertThrows(AccumuloException.class,
+ () -> ops.checkIteratorConflicts(table, iterSetting, scope));
+ ops.checkIteratorConflicts(table, iterSetting, EnumSet.of(IteratorUtil.IteratorScope.scan));
+ } finally {
+ ops.removeIterator(table, iterName, scope);
+ assertFalse(ops.listIterators(table).containsKey(iterName));
+ }
+ }
+ }
+
+ @Test
+ public void test_addConstraint_listConstraints_removeConstraint() throws Exception {
+ // addConstraint, listConstraints, removeConstraint for user tables is tested in ConstraintIT.
+ // Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(ConstraintIT.class.getName()));
+ // addConstraint, listConstraints, removeConstraint not tested for system tables. Test basic
+ // functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ var numExistingConstraints = ops.listConstraints(sysTable.tableName()).size();
+ String constraint = ComprehensiveBaseIT.TestConstraint.class.getName();
+
+ var constraintNum = ops.addConstraint(sysTable.tableName(), constraint);
+
+ var listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints + 1, listConstraints.size());
+ assertEquals(constraintNum, listConstraints.get(constraint));
+
+ ops.removeConstraint(sysTable.tableName(), constraintNum);
+
+ listConstraints = ops.listConstraints(sysTable.tableName());
+ assertEquals(numExistingConstraints, listConstraints.size());
+ assertNull(listConstraints.get(constraint));
+ }
+ }
+
+ @Test
+ public void test_getDiskUsage() throws Exception {
+ // getDiskUsage for user tables is tested in TableOperationsIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TableOperationsIT.class.getName()));
+ // getDiskUsage not tested for system tables. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ createFateTableRow(userTable);
+ createScanRefTableRow();
+ for (var sysTable : AccumuloTable.values()) {
+ ops.flush(sysTable.tableName(), null, null, true);
+ var diskUsageList = ops.getDiskUsage(Set.of(sysTable.tableName()));
+ assertEquals(1, diskUsageList.size());
+ var diskUsage = diskUsageList.get(0);
+ log.info("table : {}, disk usage : {}", sysTable.tableName(), diskUsage.getUsage());
+ if (!sysTable.equals(AccumuloTable.ROOT)) {
+ assertTrue(diskUsage.getUsage() > 0);
+ }
+ }
+ }
+
+ @Test
+ public void test_testClassLoad() throws Exception {
+ // testClassLoad is untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ // VersioningIterator is a default iterator
+ assertTrue(ops.testClassLoad(table, VersioningIterator.class.getName(),
+ SortedKeyValueIterator.class.getName()));
+ assertFalse(ops.testClassLoad(table, "foo", SortedKeyValueIterator.class.getName()));
+ }
+ }
+
+ @Test
+ public void test_setSamplerConfiguration_getSamplerConfiguration_clearSamplerConfiguration()
+ throws Exception {
+ // these are mostly untested. Test basic functionality here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+ // system and user tables
+ Set<String> allTables = ops.list();
+ for (var table : allTables) {
+ ops.setSamplerConfiguration(table, SampleIT.SC1);
+ assertEquals(SampleIT.SC1, ops.getSamplerConfiguration(table));
+ ops.clearSamplerConfiguration(table);
+ assertNull(ops.getSamplerConfiguration(table));
+ }
+ }
+
+ @Test
+ public void test_summaries_addSummarizers_removeSummarizers_listSummarizers() throws Exception {
+ // these are all tested for user tables in SummaryIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(SummaryIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ SummarizerConfiguration sc = SummarizerConfiguration.builder(BasicSummarizer.class).build();
+
+ assertThrows(AccumuloSecurityException.class,
+ () -> ops.summaries(sysTable.tableName()).retrieve());
+ ops.addSummarizers(sysTable.tableName(), sc);
+ assertEquals(List.of(sc), ops.listSummarizers(sysTable.tableName()));
+ ops.removeSummarizers(sysTable.tableName(), sc1 -> true);
+ assertTrue(ops.listSummarizers(sysTable.tableName()).isEmpty());
+ }
+ }
+
+ @Test
+ public void test_getTimeType() throws Exception {
+ // getTimeType is tested elsewhere but not for all system tables. Simple enough to test all
+ // tables here
+ userTable = getUniqueNames(1)[0];
+ ops.create(userTable);
+
+ assertEquals(TimeType.MILLIS, ops.getTimeType(userTable));
+ for (var sysTable : AccumuloTable.values()) {
+ assertEquals(TimeType.LOGICAL, ops.getTimeType(sysTable.tableName()));
+ }
+ }
+
+ @Test
+ public void test_setTabletAvailability_getTabletInformation() throws Exception {
+ // these are tested for user tables in TabletAvailabilityIT. Ensure test exists
+ assertDoesNotThrow(() -> Class.forName(TabletAvailabilityIT.class.getName()));
+ // these are not tested for system tables. Test basic functionality here
+ for (var sysTable : AccumuloTable.values()) {
+ // should not be able to unhost any system table
+ assertThrows(IllegalArgumentException.class, () -> ops
+ .setTabletAvailability(sysTable.tableName(), new Range(), TabletAvailability.UNHOSTED));
Review Comment:
Opened https://github.com/apache/accumulo/pull/5510
A lot of the table operations don't have javadoc for what the thrown exceptions mean, so maybe this can be it's own ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Comprehensive table operations IT [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5474:
URL: https://github.com/apache/accumulo/pull/5474#discussion_r2047774665
##########
core/src/main/java/org/apache/accumulo/core/clientImpl/ClientTabletCacheImpl.java:
##########
@@ -660,6 +660,7 @@ private void requestTabletHosting(ClientContext context,
}
// System tables should always be hosted
+ // TODO should this return for all sys tables?
if (AccumuloTable.ROOT.tableId() == tableId || AccumuloTable.METADATA.tableId() == tableId) {
return;
}
Review Comment:
Yes it should return for all tablets, we always expect the system tablets to be hosted. I see there was test added for trying to change their availability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org