You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/06/28 12:12:16 UTC
[GitHub] [accumulo] dlmarion opened a new pull request, #2786: Reduce merge overlap logging
dlmarion opened a new pull request, #2786:
URL: https://github.com/apache/accumulo/pull/2786
Reduce merge overlap logging by removing it from the Manager and into
the MergeInfo.overlap method. Further, only perform the logging the first
time another extent is evaluated and put into a map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion closed pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
dlmarion closed pull request #2786: Reduce merge overlap logging
URL: https://github.com/apache/accumulo/pull/2786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a diff in pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
EdColeman commented on code in PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#discussion_r908560850
##########
server/base/src/main/java/org/apache/accumulo/server/manager/state/MergeInfo.java:
##########
@@ -93,10 +100,16 @@ public boolean needsToBeChopped(KeyExtent otherExtent) {
}
public boolean overlaps(KeyExtent otherExtent) {
- boolean result = this.extent.overlaps(otherExtent);
- if (!result && needsToBeChopped(otherExtent))
- return true;
- return result;
+ return previouslyEvaluatedOverlaps.computeIfAbsent(otherExtent, b -> {
+ boolean result = this.extent.overlaps(otherExtent);
+ if (!result && needsToBeChopped(otherExtent))
+ result = true;
+ if (result)
+ LOG.debug("mergeInfo overlaps: {} true", extent);
Review Comment:
This provides additional context,which may be helpful - otherwise the extent is repeated in the log as the overlap is calculated,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
dlmarion commented on PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#issuecomment-1168990330
Related to #2788, closing as this PR won't fix this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
EdColeman commented on PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#issuecomment-1168905777
It is from my dev environment - I happen to be trying to get the FATE stack for a merge operation. So I have a hacked ShellServerIT test readily available - but no something ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
dlmarion commented on PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#issuecomment-1168897312
Is this output from a test, if so, which one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a diff in pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
EdColeman commented on code in PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#discussion_r908558104
##########
server/base/src/main/java/org/apache/accumulo/server/manager/state/MergeInfo.java:
##########
@@ -93,10 +100,16 @@ public boolean needsToBeChopped(KeyExtent otherExtent) {
}
public boolean overlaps(KeyExtent otherExtent) {
- boolean result = this.extent.overlaps(otherExtent);
- if (!result && needsToBeChopped(otherExtent))
- return true;
- return result;
+ return previouslyEvaluatedOverlaps.computeIfAbsent(otherExtent, b -> {
+ boolean result = this.extent.overlaps(otherExtent);
+ if (!result && needsToBeChopped(otherExtent))
+ result = true;
+ if (result)
+ LOG.debug("mergeInfo overlaps: {} true", extent);
Review Comment:
```suggestion
LOG.debug("mergeInfo {} overlaps: {}", otherExtent, extent);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
EdColeman commented on PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#issuecomment-1168816936
Looking at the output from the current code - I'm not sure this PR reduces any logging. The merge info is repeated in the log each time the MergeStat computed the next merge state:
The current code logs:
```2022-06-28T14:36:42,513 [accumulo.audit] INFO : operation: permitted; user: root; client: 10.113.15.42:35878; action: mergeTable; targetTable: 3; targetNamespace: 1;
2022-06-28T14:36:42,513 [manager.Manager] DEBUG: Creating merge op: 3 from startRow: -inf to endRow: +inf
2022-06-28T14:36:42,661 [manager.Manager] DEBUG: mergeInfo overlaps: 3;g< true
2022-06-28T14:36:42,662 [manager.Manager] DEBUG: mergeInfo overlaps: 3;k;g true
2022-06-28T14:36:42,664 [manager.Manager] DEBUG: mergeInfo overlaps: 3;m;k true
2022-06-28T14:36:42,665 [manager.Manager] DEBUG: mergeInfo overlaps: 3<;m true
2022-06-28T14:36:42,665 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently STARTED isDelete : false
2022-06-28T14:36:42,776 [manager.Manager] DEBUG: mergeInfo overlaps: 3;g< true
2022-06-28T14:36:42,778 [manager.Manager] DEBUG: mergeInfo overlaps: 3;k;g true
2022-06-28T14:36:42,779 [manager.Manager] DEBUG: mergeInfo overlaps: 3;m;k true
2022-06-28T14:36:42,780 [manager.Manager] DEBUG: mergeInfo overlaps: 3<;m true
2022-06-28T14:36:42,781 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
2022-06-28T14:36:43,425 [manager.Manager] DEBUG: mergeInfo overlaps: 3;g< true
2022-06-28T14:36:43,427 [manager.Manager] DEBUG: mergeInfo overlaps: 3;k;g true
2022-06-28T14:36:43,428 [manager.Manager] DEBUG: mergeInfo overlaps: 3;m;k true
2022-06-28T14:36:43,428 [manager.Manager] DEBUG: mergeInfo overlaps: 3<;m true
2022-06-28T14:36:43,429 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
2022-06-28T14:36:43,538 [manager.Manager] DEBUG: mergeInfo overlaps: 3;g< true
2022-06-28T14:36:43,538 [manager.Manager] DEBUG: mergeInfo overlaps: 3;k;g true
2022-06-28T14:36:43,538 [manager.Manager] DEBUG: mergeInfo overlaps: 3;m;k true
2022-06-28T14:36:43,539 [manager.Manager] DEBUG: mergeInfo overlaps: 3<;m true
2022-06-28T14:36:43,539 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_OFFLINE isDelete : false
2022-06-28T14:36:43,680 [manager.Manager] DEBUG: mergeInfo overlaps: 3<< true
2022-06-28T14:36:43,683 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently COMPLETE isDelete : false
2022-06-28T14:36:43,706 [merge.TableRangeOpWait] INFO : removing merge information No Merge in progress
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
EdColeman commented on PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#issuecomment-1168798143
I modified the logging statement to:
```
LOG.debug("mergeInfo {} overlaps: {}", otherExtent, extent);
```
For a small merge with 4 extents, this logs (filtered for mergeInfo):
```
2022-06-28T14:21:58,034 [manager.Manager] DEBUG: Creating merge op: 3 from startRow: -inf to endRow: +inf
2022-06-28T14:21:58,175 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:58,176 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently STARTED isDelete : false
2022-06-28T14:21:58,288 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,290 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,290 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,291 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:58,292 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
2022-06-28T14:21:58,908 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:58,911 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:58,911 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:58,912 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:58,912 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
2022-06-28T14:21:59,021 [state.MergeInfo] DEBUG: mergeInfo 3;g< overlaps: 3<<
2022-06-28T14:21:59,021 [state.MergeInfo] DEBUG: mergeInfo 3;k;g overlaps: 3<<
2022-06-28T14:21:59,022 [state.MergeInfo] DEBUG: mergeInfo 3;m;k overlaps: 3<<
2022-06-28T14:21:59,022 [state.MergeInfo] DEBUG: mergeInfo 3<;m overlaps: 3<<
2022-06-28T14:21:59,022 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_OFFLINE isDelete : false
2022-06-28T14:21:59,163 [state.MergeInfo] DEBUG: mergeInfo 3<< overlaps: 3<<
2022-06-28T14:21:59,166 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently COMPLETE isDelete : false
2022-06-28T14:21:59,178 [merge.TableRangeOpWait] INFO : removing merge information No Merge in progress
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #2786: Reduce merge overlap logging
Posted by GitBox <gi...@apache.org>.
dlmarion commented on PR #2786:
URL: https://github.com/apache/accumulo/pull/2786#issuecomment-1168923104
I don't see a clear, easy path to reduce the amount of logging. I think it will take a bunch of refactoring of merge-related code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org