You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org> on 2017/09/22 20:51:15 UTC
[kudu-CR] Prioritize freeing memory over releasing WAL segments
David Ribeiro Alves has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8128
Change subject: Prioritize freeing memory over releasing WAL segments
......................................................................
Prioritize freeing memory over releasing WAL segments
We're currently prioritizing flushing memory stores that are
anchoring log segments over memory stores that are larger.
This is problematic in situations where there are many mem stores
with a small amount of updates. In this case we'll spend a lot of
time doing small flushes that don't free that much memory potentially
allowing the server to reach memory limits more easily.
Change-Id: I0aace7c526878b139420242939462afa76d1b8df
---
M src/kudu/util/maintenance_manager.cc
1 file changed, 17 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/28/8128/1
--
To view, visit http://gerrit.cloudera.org:8080/8128
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0aace7c526878b139420242939462afa76d1b8df
Gerrit-Change-Number: 8128
Gerrit-PatchSet: 1
Gerrit-Owner: David Ribeiro Alves <da...@gmail.com>
[kudu-CR] Prioritize freeing memory over releasing WAL segments
Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has abandoned this change. ( http://gerrit.cloudera.org:8080/8128 )
Change subject: Prioritize freeing memory over releasing WAL segments
......................................................................
Abandoned
Letting this go, I don't think this is an issue.
--
To view, visit http://gerrit.cloudera.org:8080/8128
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I0aace7c526878b139420242939462afa76d1b8df
Gerrit-Change-Number: 8128
Gerrit-PatchSet: 1
Gerrit-Owner: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] Prioritize freeing memory over releasing WAL segments
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8128 )
Change subject: Prioritize freeing memory over releasing WAL segments
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8128/1/src/kudu/util/maintenance_manager.cc
File src/kudu/util/maintenance_manager.cc:
http://gerrit.cloudera.org:8080/#/c/8128/1/src/kudu/util/maintenance_manager.cc@391
PS1, Line 391: // Look at free memory. If it is dangerously low, we must select something
So the goal is to prioritize flushes of large memstores over flushes that retain the most WAL bytes? How does this change accomplish that? Seems like you've just swapped the priority of "flush largest memstore" and "GC some WAL segments". LogGC is super cheap; it's just unlinking some files on disk (hence the "low_io" prefix).
--
To view, visit http://gerrit.cloudera.org:8080/8128
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0aace7c526878b139420242939462afa76d1b8df
Gerrit-Change-Number: 8128
Gerrit-PatchSet: 1
Gerrit-Owner: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 22 Sep 2017 21:11:02 +0000
Gerrit-HasComments: Yes
[kudu-CR] Prioritize freeing memory over releasing WAL segments
Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8128 )
Change subject: Prioritize freeing memory over releasing WAL segments
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8128/1/src/kudu/util/maintenance_manager.cc
File src/kudu/util/maintenance_manager.cc:
http://gerrit.cloudera.org:8080/#/c/8128/1/src/kudu/util/maintenance_manager.cc@391
PS1, Line 391: // Look at free memory. If it is dangerously low, we must select something
> So the goal is to prioritize flushes of large memstores over flushes that r
You're right, thanks for the catch. " Look at ops that we can run quickly that free up log retention." fooled me.
--
To view, visit http://gerrit.cloudera.org:8080/8128
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0aace7c526878b139420242939462afa76d1b8df
Gerrit-Change-Number: 8128
Gerrit-PatchSet: 1
Gerrit-Owner: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 22 Sep 2017 21:47:14 +0000
Gerrit-HasComments: Yes