You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/07/30 07:36:03 UTC
[GitHub] [accumulo] EdColeman opened a new issue #1666: Improve recovery by optimizing tserver WAL processing
EdColeman opened a new issue #1666:
URL: https://github.com/apache/accumulo/issues/1666
There may be an opportunity to improve tserver performance during recovery by reading and caching the tablet markers in a wal file. The tserver might be able to read a WAL once and determine which tablets have entries and need to process the wal. Other threads could check this cache and quickly determine if the was work to do without needed to reexamine the wal file.
This issue if to investigate and see if this approach would be feasible and would provide recovery performance improvements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-786173741
Is it possible this was already completed in 1.10 with this change: https://github.com/apache/accumulo/pull/1462
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-786742784
The comments in #1462 seems to point to additional areas that could be investigated for additional improvements - this issue was generic enough to cover that - should a new issue be created to point to the possibility that additional improvements may be possible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-786373179
Closing, since it looks like this was already done. Please re-open or submit a PR if there is something more to do here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-770987478
@EdColeman - can you point me to the code that you are referring to when you say "reading and caching the tablet markers in a wal file?" Are you referring suggesting making this [map](https://github.com/apache/accumulo/blob/main/server/tserver/src/main/java/org/apache/accumulo/tserver/log/SortedLogRecovery.java#L142) a cache, or something else?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii closed issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
ctubbsii closed issue #1666:
URL: https://github.com/apache/accumulo/issues/1666
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-771051465
I'll look @keith-turner suggested this approach - from what I recall, during recovery the logs would show repeated messages from reading / processing the wals.
I think it may have been the list of wals was processed multiple times - once for each tablet? I believe the idea was that the processing could occur once and then be cached for the duration of that recovery session. I didn't explore if that cache should be a map or another in-memory representation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-786173967
This maybe be OBE with #1462 - but in 1.9.3, the error message here - https://github.com/apache/accumulo/blob/a834aa8f36f1305b3d41abccb4eadc4f3b39302c/server/master/src/main/java/org/apache/accumulo/master/recovery/RecoveryManager.java#L156 will create a wall of log statements during recovery (in the master debug log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #1666: Improve recovery by optimizing tserver WAL processing
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #1666:
URL: https://github.com/apache/accumulo/issues/1666#issuecomment-786816005
> The comments in #1462 seems to point to additional areas that could be investigated for additional improvements - this issue was generic enough to cover that - should a new issue be created to point to the possibility that additional improvements may be possible?
Issues work best when they are narrowly targeted and actionable, so somebody can grab one from the queue and work on it. I didn't see anything like that in the referenced PR, but if you do, please feel free to create a new issue to highlight and track the specific work to be done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org