You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/08/18 08:11:10 UTC
[GitHub] [incubator-doris] zenoyang opened a new pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
zenoyang opened a new pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448
## Proposed changes
fix https://github.com/apache/incubator-doris/issues/6447
1. FE master regularly triggers the remove operation
2. After the master completes the removal of deleteInfo, it is synchronized to the Follower through editlog for remove
3. When the DeleteInfo creation time is longer than the current time, it will be cleaned up, which is determined by the `delete_info_keep_max_second` configuration
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [x] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
- [x] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] If these changes need document changes, I have updated the document
- [x] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-906167308
> Your implementation can ensure consistency, but this approach is a bit complicated. We should minimize unnecessary state maintenance.
>
> So I suggest to follow the logic of `labelCleaner`, and each FE cleans up its own deleteInfo. Regarding the issue of consistency, you can forward the `show delete` statement to the master FE for execution, so that we can get consistent results.
>
> Regarding the forwarding logic, you can refer to the `getRedirectStatus` method of `ShowLoadStmt.java`
Thank you Mingyu for your suggestion. I have submitted a new version of the code: each FE cleans up own deleteInfo, and forwards the show delete statement to the master FE for execution. Please review it again, thank you everyone ^_^ @morningman @xy720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900910166
> Hi @zenoyang thanks for your PR. This is indeed a problem.
> But there some thing that you need to know:
>
> 1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
> 2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
>
> This logic is a bit convoluted. If you don't understand something, welcome to communicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693541872
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
+ return deleteHandler;
+ }
+
+ /**
+ * FE Master timing, or when any FE replay meta triggers DeleteInfo cleanup
+ */
+ public void removeOldDeleteInfos() {
+ Timestamp timestamp = new Timestamp();
+ removeOldDeleteInfos(timestamp);
+ Catalog.getCurrentCatalog().getEditLog().logRemoveOldDeleteInfos(timestamp);
Review comment:
It has been repaired and completed the test, please review it again, thank you very much : ) @xy720
When the master or follower FE loads the image when it starts, old deleteInfos will not clean up and generate editlog.
FE Master timing or checkpoint loading image, trigger DeleteInfo cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693399525
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
Review comment:
Hi, there is problem in here. When we are loading image, we shouldn't write the editLog, or else replayRemoveOldDeleteInfos() method will be triggered immediately after loading image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693399525
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
Review comment:
Hi, there is problem in here. When we are load image, we shouldn't write the editLog, or else replayRemoveOldDeleteInfos() method will triggered after loading image immediately.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
+ return deleteHandler;
+ }
+
+ /**
+ * FE Master timing, or when any FE replay meta triggers DeleteInfo cleanup
+ */
+ public void removeOldDeleteInfos() {
+ Timestamp timestamp = new Timestamp();
+ removeOldDeleteInfos(timestamp);
+ Catalog.getCurrentCatalog().getEditLog().logRemoveOldDeleteInfos(timestamp);
Review comment:
I think you can add a judgment here that when Fe is loading image, it will not execute this line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r696721817
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -1340,6 +1340,8 @@ private void startNonMasterDaemonThreads() {
esRepository.start();
// domain resolver
domainResolver.start();
+ // start daemon thread to clean up old deleteInfos
+ deleteHandler.start();
Review comment:
No need to start a new thread to do this.
You can just call `deleteHandler.removeOldDeleteInfos()` in `labelCleaner` in `Catalog`
```
public void createLabelCleaner() {
labelCleaner = new MasterDaemon("LoadLabelCleaner", Config.label_clean_interval_second * 1000L) {
@Override
protected void runAfterCatalogReady() {
load.removeOldLoadJobs();
loadManager.removeOldLoadJob();
exportMgr.removeOldExportJobs();
deleteHandler.removeOldDeleteInfos();
}
};
}
```
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +746,48 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
+ return deleteHandler;
+ }
+
+ /**
+ * All FE timings or Master FE checkpoint trigger the deletion of old deleteInfos operations
+ */
+ public void removeOldDeleteInfos() {
+ if (Catalog.getCurrentCatalog().isReady()) {
Review comment:
No need to check `Catalog.getCurrentCatalog().isReady()` now.
And you can simply merge the 2 `removeOldDeleteInfos()` method.
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -153,6 +153,19 @@
*/
@ConfField public static int label_clean_interval_second = 4 * 3600; // 4 hours
+ /**
+ * Delete all deleteInfo older than *delete_info_keep_max_second*
+ * Setting a shorter time will reduce FE memory usage and image file size
+ * (Because all deleteInfo is stored in memory and image files before being deleted)
+ */
+ @ConfField(mutable = true)
+ public static int delete_info_keep_max_second = 3 * 24 * 3600; // 3 days
+
+ /**
+ * DeleteHandler will run every *delete_info_clean_interval_second* to clean old delete infos.
+ */
+ @ConfField public static int delete_info_clean_interval_second = 3600; // 1 hours
Review comment:
This can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang edited a comment on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang edited a comment on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900910166
> Hi @zenoyang thanks for your PR. This is indeed a problem.
> But there some thing that you need to know:
>
> 1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
> 2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
>
> This logic is a bit convoluted. If you don't understand something, welcome to communicate.
Thank you very much for @morningman 's review, I really didn't consider it thoroughly before. Now I have sorted out the new plan.
If the EditLog synchronization logic is canceled, each FE regularly cleans up DeleteInfo, which is relatively simple to implement, but the currentTimeMs of each FE is different when cleaning, so there will be inconsistencies in DeleteInfo in each FE, although the inconsistency seems to have little effect.
So my plan is:
- Execute regularly on the Master FE, and notify other FEs to execute through EditLog, all FEs sharing the same timestamp will not cause inconsistencies.
- When an FE plays back metadata, after the deleteHandler is initialized, the DeleteInfo cleanup logic is triggered, and other FEs are notified to clean up through EditLog to ensure consistency.
I hope you can help to see if there are any problems with this, thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900447517
Hi @zenoyang thanks for your PR. This is indeed a problem.
But there some thing that you need to know:
1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
This logic is a bit convoluted. If you don't understand something, welcome to communicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang closed pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang closed pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-907598257
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang edited a comment on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang edited a comment on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900910166
> Hi @zenoyang thanks for your PR. This is indeed a problem.
> But there some thing that you need to know:
>
> 1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
> 2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
>
> This logic is a bit convoluted. If you don't understand something, welcome to communicate.
Thank you very much for @morningman 's review, I really didn't consider it thoroughly before. Now I have sorted out the new plan.
If the EditLog synchronization logic is canceled, each FE regularly cleans up DeleteInfo, which is relatively simple to implement, but the currentTimeMs of each FE is different when cleaning, so there will be inconsistencies in DeleteInfo in each FE, although the inconsistency seems to have little effect.
So my plan is:
- Execute regularly on the Master FE, and notify other FEs to execute through EditLog, all FEs sharing the same timestamp will not cause inconsistencies.
- When an FE plays back metadata, after the deleteHandler is initialized, the DeleteInfo cleanup logic is triggered, and other FEs are notified to clean up through EditLog to ensure consistency.
I hope you can help to see if there are any problems with this, thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693541872
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
+ return deleteHandler;
+ }
+
+ /**
+ * FE Master timing, or when any FE replay meta triggers DeleteInfo cleanup
+ */
+ public void removeOldDeleteInfos() {
+ Timestamp timestamp = new Timestamp();
+ removeOldDeleteInfos(timestamp);
+ Catalog.getCurrentCatalog().getEditLog().logRemoveOldDeleteInfos(timestamp);
Review comment:
It has been fixed, please review it again, thank you very much : )
When the master or follower FE loads the image when it starts, old deleteInfos will not clean up and generate editlog.
FE Master timing or checkpoint loading image, trigger DeleteInfo cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang closed pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang closed pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693399525
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
Review comment:
Hi, there is problem in here. When we are loading image, we shouldn't write the editLog, or else some error will report (Because bdbje is open only after the image has been loaded).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r696757229
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -1340,6 +1340,8 @@ private void startNonMasterDaemonThreads() {
esRepository.start();
// domain resolver
domainResolver.start();
+ // start daemon thread to clean up old deleteInfos
+ deleteHandler.start();
Review comment:
already fixed
##########
File path: fe/fe-core/src/main/java/org/apache/doris/common/Config.java
##########
@@ -153,6 +153,19 @@
*/
@ConfField public static int label_clean_interval_second = 4 * 3600; // 4 hours
+ /**
+ * Delete all deleteInfo older than *delete_info_keep_max_second*
+ * Setting a shorter time will reduce FE memory usage and image file size
+ * (Because all deleteInfo is stored in memory and image files before being deleted)
+ */
+ @ConfField(mutable = true)
+ public static int delete_info_keep_max_second = 3 * 24 * 3600; // 3 days
+
+ /**
+ * DeleteHandler will run every *delete_info_clean_interval_second* to clean old delete infos.
+ */
+ @ConfField public static int delete_info_clean_interval_second = 3600; // 1 hours
Review comment:
already fixed
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +746,48 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
+ return deleteHandler;
+ }
+
+ /**
+ * All FE timings or Master FE checkpoint trigger the deletion of old deleteInfos operations
+ */
+ public void removeOldDeleteInfos() {
+ if (Catalog.getCurrentCatalog().isReady()) {
Review comment:
already fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693399525
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
Review comment:
Hi, there is problem in here. When we are load image, we shouldn't write the editLog, or else replayRemoveOldDeleteInfos() method will be triggered after loading image immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-905333265
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900447517
Hi @zenoyang thanks for your PR. This is indeed a problem.
But there some thing that you need to know:
1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
This logic is a bit convoluted. If you don't understand something, welcome to communicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693541872
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
+ return deleteHandler;
+ }
+
+ /**
+ * FE Master timing, or when any FE replay meta triggers DeleteInfo cleanup
+ */
+ public void removeOldDeleteInfos() {
+ Timestamp timestamp = new Timestamp();
+ removeOldDeleteInfos(timestamp);
+ Catalog.getCurrentCatalog().getEditLog().logRemoveOldDeleteInfos(timestamp);
Review comment:
It has been fixed, please review it again, thank you very much : ) @xy720
When the master or follower FE loads the image when it starts, old deleteInfos will not clean up and generate editlog.
FE Master timing or checkpoint loading image, trigger DeleteInfo cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#discussion_r693399525
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java
##########
@@ -709,6 +745,51 @@ public void write(DataOutput out) throws IOException {
public static DeleteHandler read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ DeleteHandler deleteHandler = GsonUtils.GSON.fromJson(json, DeleteHandler.class);
+ deleteHandler.removeOldDeleteInfos();
Review comment:
Hi, there is problem in here. When we are load image, we shouldn't write the editLog, or else replayRemoveOldDeleteInfos() method will be triggered immediately after loading image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-901875095
I submitted a new code to fix the previous problem. Please review it when you have time. ^_^ @morningman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on pull request #6448: [Bug] Regularly clean up old DeleteInfos in the DeleteHandler
Posted by GitBox <gi...@apache.org>.
zenoyang commented on pull request #6448:
URL: https://github.com/apache/incubator-doris/pull/6448#issuecomment-900910166
> Hi @zenoyang thanks for your PR. This is indeed a problem.
> But there some thing that you need to know:
>
> 1. The `labelCleaner` thread is running on all FEs, each FE will clean its own `deleteInfos`. So don't need to write a edit log `OP_REMOVE_OLD_DELETE_INFO` to "tell" other FE to clean the delete infos.
> 2. Once the `labelCleaner` thread cleans the delete infos on each FE. What we need to take care is to clean the delete infos in meta replay logic. Eg, after calling `DeleteHandler.read()` method, we should clean the delete info actively, so that the meta image won't contains expired delete infos.
>
> This logic is a bit convoluted. If you don't understand something, welcome to communicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org