You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2018/02/14 20:51:14 UTC
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Hello Kudu Jenkins, Todd Lipcon,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/9322
to review the following change.
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
KUDU-2238. DMS not flush under memory pressure.
When we choose DMS to flush, now we always pick the DMS with
highest log retention. However, as KUDU-2238 shows, in some cases
DMS with highest log retention may only consume little memory, and
other DMSs may consume much more memory, but get no chance to be
flushed, even under memory pressure.
This patch gives the ability to take memory consumption into
consideration when we choose DMS.
Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Reviewed-on: http://gerrit.cloudera.org:8080/8904
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_replica_mm_ops.cc
3 files changed, 26 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/22/9322/1
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, zhen.zhang, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9322
to look at the new patch set (#2).
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
KUDU-2238. DMS not flush under memory pressure.
When we choose DMS to flush, now we always pick the DMS with
highest log retention. However, as KUDU-2238 shows, in some cases
DMS with highest log retention may only consume little memory, and
other DMSs may consume much more memory, but get no chance to be
flushed, even under memory pressure.
This patch gives the ability to take memory consumption into
consideration when we choose DMS.
Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Reviewed-on: http://gerrit.cloudera.org:8080/8904
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_replica_mm_ops.cc
3 files changed, 18 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/22/9322/2
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/9322 )
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
Patch Set 2:
> Build Failed
>
> http://jenkins.kudu.apache.org/job/kudu-gerrit/12030/ : FAILURE
Seems like the python tests are failing because the build slave is too old?
"your setuptools is too old (<12)
setuptools_scm functionality is degraded"
Then after that the build outputs for debug v release are very different, and it doesn;t seem the python tests even run in debug though there's no error message beyond an assertion the tests failed.
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>
Gerrit-Comment-Date: Fri, 16 Feb 2018 16:07:36 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9322 )
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
KUDU-2238. DMS not flush under memory pressure.
When we choose DMS to flush, now we always pick the DMS with
highest log retention. However, as KUDU-2238 shows, in some cases
DMS with highest log retention may only consume little memory, and
other DMSs may consume much more memory, but get no chance to be
flushed, even under memory pressure.
This patch gives the ability to take memory consumption into
consideration when we choose DMS.
Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Reviewed-on: http://gerrit.cloudera.org:8080/8904
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/9322
Tested-by: Will Berkeley <wd...@gmail.com>
Reviewed-by: Will Berkeley <wd...@gmail.com>
---
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_replica_mm_ops.cc
3 files changed, 18 insertions(+), 7 deletions(-)
Approvals:
Will Berkeley: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/9322 )
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>
Gerrit-Comment-Date: Thu, 22 Feb 2018 21:23:09 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/9322 )
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
Patch Set 2: Verified+1
The python build failures are due to an old setuptools:
Using KUDU_HOME directory: /home/jenkins-slave/workspace/kudu-branch-1.5.x/1
Using KUDU_BUILD directory: /home/jenkins-slave/workspace/kudu-branch-1.5.x/1/build/debug
/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
your setuptools is too old (<12)
The tests pass on release/asan/tsan consistently, just not debug, with no sign the tests actually run.
The tsan failure in the latest in an unrelated bug (also fixed in later versions)
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>
Gerrit-Comment-Date: Thu, 22 Feb 2018 21:22:11 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.5.x) KUDU-2238. DMS not flush under memory pressure.
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/9322 )
Change subject: KUDU-2238. DMS not flush under memory pressure.
......................................................................
Removed reviewer Kudu Jenkins with the following votes:
* Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/9322
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.5.x
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I0c04d76aa0e3888352dc56eeb493a5437ef47e42
Gerrit-Change-Number: 9322
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: zhen.zhang <zh...@xiaomi.com>