You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2023/07/09 00:57:59 UTC
[GitHub] [accumulo] keith-turner opened a new pull request, #3578: Makes user compaction metadata independent.
keith-turner opened a new pull request, #3578:
URL: https://github.com/apache/accumulo/pull/3578
Before this change user initiated compactions stored config in a single place in zookeeper and had a single counter in each tablet. This system made it impossible for concurrent user initiated compactions to run on the same table with different configuration. This commit adds compaction config storage and tablet metadata that has 1:1 correspondence with the fate operation driving a user initiated compaction. This allows each user initiated compaction to be independent. This new system also avoids some odd race conditions with tablet metadata updates that existed with old per tablet compaction counter. Instead of a counter each tablet now conceptually has a set of the fate transaction ids that have completed a compaction.
See the new documention in TableOperations.compact() for an example.
fixes #3517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on pull request #3578: Makes user compaction metadata independent.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #3578:
URL: https://github.com/apache/accumulo/pull/3578#issuecomment-1629217192
This commit also implements compaction cancellation and fixes a few misc compaction bugs in the elasticity branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner merged pull request #3578: Makes user compaction metadata independent.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #3578:
URL: https://github.com/apache/accumulo/pull/3578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on pull request #3578: Makes user compaction metadata independent.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #3578:
URL: https://github.com/apache/accumulo/pull/3578#issuecomment-1629496581
All tests in CompactionIT are passing with these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a diff in pull request #3578: Makes user compaction metadata independent.
Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #3578:
URL: https://github.com/apache/accumulo/pull/3578#discussion_r1261443103
##########
core/src/main/java/org/apache/accumulo/core/client/admin/TableOperations.java:
##########
@@ -305,8 +305,23 @@ void compact(String tableName, Text start, Text end, List<IteratorSetting> itera
* </ul>
*
* <p>
- * If two threads call this method concurrently for the same table and set one or more of the
- * above then one thread will fail.
+ * Starting with Accumulo 4.0 concurrent compactions can be initiated on a table with different
+ * configuration. Prior to 4.0 if this were done, then only one would work and the others would
+ * throw an exception. When concurrent compactions with different configuration run each tablet
+ * will be compacted once for each user initiated compaction in some arbitrary order. For example
+ * consider the following situation.
+ *
+ * <OL>
+ * <LI>Table A has three tablets Tab1, Tab2, Tab3</LI>
+ * <LI>This method is called to initiate a compaction on Tablets Tab1 and Tab2 with iterators
+ * I1</LI>
+ * <LI>This method is called to initiate a compaction on Tablets Tab2 and Tab3 with iterators
+ * I2</LI>
+ * <LI>Tablet Tab1 will compact with iterator I1</LI>
+ * <LI>Two compactions will happen for tablet Tab2. It will either compact with iterators I1 and
+ * then I2 OR it will compact with iterators I2 and then I1.</LI>
+ * <LI>Tablet Tab3 will compact with iterator I2</LI>
+ * </OL>
Review Comment:
```suggestion
* Starting with Accumulo, 4.0 concurrent compactions can be initiated on a table with different
* configuration. Prior to 4.0, if this were done, then only one compaction would work and the
* others would throw an exception. When concurrent compactions with different configuration run,
* each tablet will be compacted once for each user initiated compaction in some arbitrary order.
* For example consider the following situation.
*
* <ol>
* <li>Table A has three tablets Tab1, Tab2, Tab3</li>
* <li>This method is called to initiate a compaction on Tablets Tab1 and Tab2 with iterator
* I1</li>
* <li>This method is called to initiate a compaction on Tablets Tab2 and Tab3 with iterator
* I2</li>
* <li>Tablet Tab1 will compact with iterator I1</li>
* <li>Two compactions will happen for tablet Tab2. It will either compact with iterator I1 and
* then I2 OR it will compact with iterator I2 and then I1.</li>
* <li>Tablet Tab3 will compact with iterator I2</li>
* </ol>
```
Made the tags lowercase and tried to improve the wording a tiny bit in places. Also formatted after changing.
##########
core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java:
##########
@@ -378,6 +377,15 @@ public static class ExternalCompactionColumnFamily {
public static final Text NAME = new Text(STR_NAME);
}
+ /**
+ * This is family is used to track which tablets were compacted by a user compaction. The column
Review Comment:
```suggestion
* This family is used to track which tablets were compacted by a user compaction. The column
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on pull request #3578: Makes user compaction metadata independent.
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #3578:
URL: https://github.com/apache/accumulo/pull/3578#issuecomment-1633355858
Merged this because I have another PR that builds on this to remove code from the tserver and I need to get that change in so I can start making changes to improve the tablet server code. If anyone was looking at this feel free to comment still I can address any comments in follow on commits or issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org