You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2024/02/29 16:52:34 UTC
[PR] Considers all tablet metadata columns in split code [accumulo]
keith-turner opened a new pull request, #4323:
URL: https://github.com/apache/accumulo/pull/4323
Made the following changes to the split code that adds new tablets and updates the existing tablet.
* fixed potential NPE w/ tablet operation id check by reversing order of equals check
* Throws IllegalStateException when attempting to split tablet with merged or cloned markers
* Removed adding wals when creating new tablets in split, its not expected that the parent tablet would have wals and this is checked earlier
* Deleted any user compaction requested, hosting requested, suspended, or last columns in the parent tablet
Added a unit test that attempts to exercise the split code with all tablet columns. The unit test also has a set of tablet columns that were verified to work with split and it is checked against the set of columns in the code. The purpose of this test is to fail when a new column is added to ensure that split is considered.
Was a bit uncertain about deleting the last location and suspend. Those columns either need to be deleted from the parent tablet or added to the new tablets being created. The current code was doing neither. Decided to delete them as the new tablets have a different range and are conceptually different tablets than the parent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #4323:
URL: https://github.com/apache/accumulo/pull/4323#discussion_r1509286816
##########
server/manager/src/test/java/org/apache/accumulo/manager/tableOps/split/UpdateTabletsTest.java:
##########
@@ -47,6 +75,33 @@ FileUtil.FileInfo newFileInfo(String start, String end) {
return new FileUtil.FileInfo(new Text(start), new Text(end));
}
+ /**
+ * This is a set of tablet metadata columns that the split code is known to handle. The purpose of
+ * the set is to detect when a new tablet metadata column was added without considering the
+ * implications for splitting tablets. For a column to be in this set it means an Accumulo
+ * developer has determined that split code can handle that column OR has opened an issue about
+ * handling it.
+ */
+ private static final Set<ColumnType> COLUMNS_HANDLED_BY_SPLIT =
+ EnumSet.of(ColumnType.TIME, ColumnType.LOGS, ColumnType.FILES, ColumnType.PREV_ROW,
+ ColumnType.OPID, ColumnType.LOCATION, ColumnType.ECOMP, ColumnType.SELECTED,
+ ColumnType.LOADED, ColumnType.USER_COMPACTION_REQUESTED, ColumnType.MERGED,
+ ColumnType.LAST, ColumnType.SCANS, ColumnType.DIR, ColumnType.CLONED, ColumnType.FLUSH_ID,
+ ColumnType.FLUSH_NONCE, ColumnType.SUSPEND, ColumnType.AVAILABILITY,
+ ColumnType.HOSTING_REQUESTED, ColumnType.COMPACTED);
+
+ /**
+ * The purpose of this test is to catch new tablet metadata columns that were added w/o
+ * considering splitting tablets.
+ */
+ @Test
+ public void checkColumns() {
+ for (ColumnType columnType : ColumnType.values()) {
+ assertTrue(COLUMNS_HANDLED_BY_SPLIT.contains(columnType),
Review Comment:
I don't know if this is easily possible with EasyMock but I was wondering if the mock API supports a way to check that all methods have been mocked with expectations? I suppose we could just use reflection to iterate over the methods and check each one, but I'm curious if there's just a simpler way to check the mocked object has methods that haven't been mocked so we could throw an error if new methods showed up but were not mocked with a return value, which would of course imply it was not considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #4323:
URL: https://github.com/apache/accumulo/pull/4323#discussion_r1509286816
##########
server/manager/src/test/java/org/apache/accumulo/manager/tableOps/split/UpdateTabletsTest.java:
##########
@@ -47,6 +75,33 @@ FileUtil.FileInfo newFileInfo(String start, String end) {
return new FileUtil.FileInfo(new Text(start), new Text(end));
}
+ /**
+ * This is a set of tablet metadata columns that the split code is known to handle. The purpose of
+ * the set is to detect when a new tablet metadata column was added without considering the
+ * implications for splitting tablets. For a column to be in this set it means an Accumulo
+ * developer has determined that split code can handle that column OR has opened an issue about
+ * handling it.
+ */
+ private static final Set<ColumnType> COLUMNS_HANDLED_BY_SPLIT =
+ EnumSet.of(ColumnType.TIME, ColumnType.LOGS, ColumnType.FILES, ColumnType.PREV_ROW,
+ ColumnType.OPID, ColumnType.LOCATION, ColumnType.ECOMP, ColumnType.SELECTED,
+ ColumnType.LOADED, ColumnType.USER_COMPACTION_REQUESTED, ColumnType.MERGED,
+ ColumnType.LAST, ColumnType.SCANS, ColumnType.DIR, ColumnType.CLONED, ColumnType.FLUSH_ID,
+ ColumnType.FLUSH_NONCE, ColumnType.SUSPEND, ColumnType.AVAILABILITY,
+ ColumnType.HOSTING_REQUESTED, ColumnType.COMPACTED);
+
+ /**
+ * The purpose of this test is to catch new tablet metadata columns that were added w/o
+ * considering splitting tablets.
+ */
+ @Test
+ public void checkColumns() {
+ for (ColumnType columnType : ColumnType.values()) {
+ assertTrue(COLUMNS_HANDLED_BY_SPLIT.contains(columnType),
Review Comment:
I don't know if this is easily possible with EasyMock but I was wondering if the mock API supports a way to check that all methods have been mocked with expectations? I suppose we could just use reflection to iterate over the methods and check each one, but I'm curious if there's a simpler way to check the mocked object has methods that haven't been mocked so we could throw an error if new methods showed up but were not mocked with a return value, which would of course imply it was not considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #4323:
URL: https://github.com/apache/accumulo/pull/4323#discussion_r1510019802
##########
server/manager/src/test/java/org/apache/accumulo/manager/tableOps/split/UpdateTabletsTest.java:
##########
@@ -47,6 +75,33 @@ FileUtil.FileInfo newFileInfo(String start, String end) {
return new FileUtil.FileInfo(new Text(start), new Text(end));
}
+ /**
+ * This is a set of tablet metadata columns that the split code is known to handle. The purpose of
+ * the set is to detect when a new tablet metadata column was added without considering the
+ * implications for splitting tablets. For a column to be in this set it means an Accumulo
+ * developer has determined that split code can handle that column OR has opened an issue about
+ * handling it.
+ */
+ private static final Set<ColumnType> COLUMNS_HANDLED_BY_SPLIT =
+ EnumSet.of(ColumnType.TIME, ColumnType.LOGS, ColumnType.FILES, ColumnType.PREV_ROW,
+ ColumnType.OPID, ColumnType.LOCATION, ColumnType.ECOMP, ColumnType.SELECTED,
+ ColumnType.LOADED, ColumnType.USER_COMPACTION_REQUESTED, ColumnType.MERGED,
+ ColumnType.LAST, ColumnType.SCANS, ColumnType.DIR, ColumnType.CLONED, ColumnType.FLUSH_ID,
+ ColumnType.FLUSH_NONCE, ColumnType.SUSPEND, ColumnType.AVAILABILITY,
+ ColumnType.HOSTING_REQUESTED, ColumnType.COMPACTED);
+
+ /**
+ * The purpose of this test is to catch new tablet metadata columns that were added w/o
+ * considering splitting tablets.
+ */
+ @Test
+ public void checkColumns() {
+ for (ColumnType columnType : ColumnType.values()) {
+ assertTrue(COLUMNS_HANDLED_BY_SPLIT.contains(columnType),
Review Comment:
> I don't know if this is easily possible with EasyMock but I was wondering if the mock API supports a way to check that all methods have been mocked with expectations?
For this case I was kinda covering it with that set of columns reviewed by a developer in the test. Not sure how to do that in EasyMock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #4323:
URL: https://github.com/apache/accumulo/pull/4323#discussion_r1509286816
##########
server/manager/src/test/java/org/apache/accumulo/manager/tableOps/split/UpdateTabletsTest.java:
##########
@@ -47,6 +75,33 @@ FileUtil.FileInfo newFileInfo(String start, String end) {
return new FileUtil.FileInfo(new Text(start), new Text(end));
}
+ /**
+ * This is a set of tablet metadata columns that the split code is known to handle. The purpose of
+ * the set is to detect when a new tablet metadata column was added without considering the
+ * implications for splitting tablets. For a column to be in this set it means an Accumulo
+ * developer has determined that split code can handle that column OR has opened an issue about
+ * handling it.
+ */
+ private static final Set<ColumnType> COLUMNS_HANDLED_BY_SPLIT =
+ EnumSet.of(ColumnType.TIME, ColumnType.LOGS, ColumnType.FILES, ColumnType.PREV_ROW,
+ ColumnType.OPID, ColumnType.LOCATION, ColumnType.ECOMP, ColumnType.SELECTED,
+ ColumnType.LOADED, ColumnType.USER_COMPACTION_REQUESTED, ColumnType.MERGED,
+ ColumnType.LAST, ColumnType.SCANS, ColumnType.DIR, ColumnType.CLONED, ColumnType.FLUSH_ID,
+ ColumnType.FLUSH_NONCE, ColumnType.SUSPEND, ColumnType.AVAILABILITY,
+ ColumnType.HOSTING_REQUESTED, ColumnType.COMPACTED);
+
+ /**
+ * The purpose of this test is to catch new tablet metadata columns that were added w/o
+ * considering splitting tablets.
+ */
+ @Test
+ public void checkColumns() {
+ for (ColumnType columnType : ColumnType.values()) {
+ assertTrue(COLUMNS_HANDLED_BY_SPLIT.contains(columnType),
Review Comment:
I don't know if this is easily possible with EasyMock but I was wondering if the mock API supports a way to check that all methods have been mocked with expectations? I suppose just use reflection to iterate over the methods and check each one, but I'm curious if there's just a simpler way to check the mocked object has methods that haven't been mocked so we could throw an error if new methods showed up but were not mocked with a return value, which would of course imply it was not considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #4323:
URL: https://github.com/apache/accumulo/pull/4323#issuecomment-1973556685
This is probably something to do as a follow on when the merge test is looked at, but you could try to move common code related to mocking TabletMetadata to a central place like a utility class to be reused. My assumption is a lot of setup for all of the mocks could be re-used for testing that merge handles all the columns. It should make it easier easier to update the tests when adding new columns that need to be considered for both split and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #4323:
URL: https://github.com/apache/accumulo/pull/4323#issuecomment-1973712131
> I'm wondering if we could get into a weird state where we just start rapidly throwing exceptions in a loop as it tries to split over and over and keeps failing so it would be good to test what the outcome is if there's unexpected columns that show up and that it's acceptable behavior for the system.
It would be good to add an IT for this. Could manually add an unexpected column to a tablet in an IT and then attempt to split it. I'll add that in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Considers all tablet metadata columns in split code [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #4323:
URL: https://github.com/apache/accumulo/pull/4323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org