You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2017/08/10 22:49:43 UTC
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Tim Armstrong has uploaded a new change for review.
http://gerrit.cloudera.org:8080/7653
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
IMPALA-5677: limit clean page memory consumption
Adds the following flag:
-buffer_pool_clean_pages_limit ((Advanced) Limit on bytes of clean spilled
pages that will be accumulated in the buffer pool. Specified as number of
bytes ('<int>[bB]?'), megabytes ('<float>[mM]'), gigabytes
('<float>[gG]'), or percentage of the buffer pool limit ('<int>%').
Defaults to bytes if no unit is given..) type: string default: "10%"
This helps prevent Impala accumulating excessive amounts of clean pages,
which can cause some problems in practice:
* The OS buffer cache is squeezed, reducing I/O performance from HDFS
and potentially reducing the ability of the OS to absorb writes from
Impala without blocking.
* Impala process memory consumption can expand more than users or admins
might expect. E.g. if one query is running with a mem_limit of 1GB,
many people will be surprised if the process inflates to the full
process limit of 100GB. Impala doesn't provide any guarantees except
from staying within the process mem_limit, but this could be a
surprising divergence from past behaviour.
Observability:
A new metric buffer-pool.clean-pages-limit is added.
Testing:
Added a backend test to directly test that clean pages are evicted.
Ran in a loop to flush out any flakiness.
Ran exhaustive tests.
Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
---
M be/src/common/global-flags.cc
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/test-env.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 234 insertions(+), 72 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/7653/1
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 5: Code-Review+2
rebase
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/7653
to look at the new patch set (#6).
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
IMPALA-5677: limit clean page memory consumption
Adds the following flag:
-buffer_pool_clean_pages_limit ((Advanced) Limit on bytes of clean spilled
pages that will be accumulated in the buffer pool. Specified as number of
bytes ('<int>[bB]?'), megabytes ('<float>[mM]'), gigabytes
('<float>[gG]'), or percentage of the buffer pool limit ('<int>%').
Defaults to bytes if no unit is given..) type: string default: "10%"
This helps prevent Impala accumulating excessive amounts of clean pages,
which can cause some problems in practice:
* The OS buffer cache is squeezed, reducing I/O performance from HDFS
and potentially reducing the ability of the OS to absorb writes from
Impala without blocking.
* Impala process memory consumption can expand more than users or admins
might expect. E.g. if one query is running with a mem_limit of 1GB,
many people will be surprised if the process inflates to the full
process limit of 100GB. Impala doesn't provide any guarantees except
from staying within the process mem_limit, but this could be a
surprising divergence from past behaviour.
Observability:
A new metric buffer-pool.clean-pages-limit is added.
Testing:
Added a backend test to directly test that clean pages are evicted.
Ran in a loop to flush out any flakiness.
Ran exhaustive tests.
Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
---
M be/src/common/global-flags.cc
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/test-env.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 241 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/7653/6
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/7653/1//COMMIT_MSG
Commit Message:
Line 34: Ran in a loop to flush out any flakiness.
> do we explicitly test the clean_page_limit={0, buffer_pool_limit} cases?
I don't have tests for those. We could add a backend test for those cases. There isn't a distinct code path for those cases but I guess it's good to test the edge cases.
http://gerrit.cloudera.org:8080/#/c/7653/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
PS1, Line 61: spilled
> maybe say "unpinned", since "spilled" is more of a operator concept rather
Done
http://gerrit.cloudera.org:8080/#/c/7653/1/be/src/runtime/bufferpool/buffer-allocator.cc
File be/src/runtime/bufferpool/buffer-allocator.cc:
Line 109: int64_t GetCleanPageBytes();
> is that still used?
Done
PS1, Line 636: one page
> what does that mean? maybe say, "no other page to evict"?
Done
http://gerrit.cloudera.org:8080/#/c/7653/1/be/src/runtime/bufferpool/buffer-allocator.h
File be/src/runtime/bufferpool/buffer-allocator.h:
PS1, Line 222: the
> they?
Done
PS1, Line 225: remaining
> I'm not sure it's clear what it means to be "remaining".
Reworded the comment.
PS1, Line 228: clean_page_bytes_remaining_
> also, this name might be a little confusing. It's nice that it's consistent
Documented that this is clean_page_bytes_limit_ - bytes of clean pages. I think that makes it less ambiguous.
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1089/
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/7653/1//COMMIT_MSG
Commit Message:
Line 34: Ran in a loop to flush out any flakiness.
do we explicitly test the clean_page_limit={0, buffer_pool_limit} cases?
http://gerrit.cloudera.org:8080/#/c/7653/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
PS1, Line 61: spilled
maybe say "unpinned", since "spilled" is more of a operator concept rather than a buffer pool concept. Or just delete the word since clean implies it.
http://gerrit.cloudera.org:8080/#/c/7653/1/be/src/runtime/bufferpool/buffer-allocator.cc
File be/src/runtime/bufferpool/buffer-allocator.cc:
Line 109: int64_t GetCleanPageBytes();
is that still used?
PS1, Line 636: one page
what does that mean? maybe say, "no other page to evict"?
http://gerrit.cloudera.org:8080/#/c/7653/1/be/src/runtime/bufferpool/buffer-allocator.h
File be/src/runtime/bufferpool/buffer-allocator.h:
PS1, Line 222: the
they?
PS1, Line 225: remaining
I'm not sure it's clear what it means to be "remaining".
PS1, Line 228: clean_page_bytes_remaining_
also, this name might be a little confusing. It's nice that it's consistent with system_bytes_remaining_, but I think it's a bit less intuitive since it's not "remaining" in the same sense -- i.e. it's not tracking bytes of clean pages. But maybe just updating the comment will be sufficient. Maybe also saying what the equation is, i.e.:
clean_page_bytes_reamining_ + bytes currently used by clean pages = clean_page_bytes_limit_
would help?
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
IMPALA-5677: limit clean page memory consumption
Adds the following flag:
-buffer_pool_clean_pages_limit ((Advanced) Limit on bytes of clean spilled
pages that will be accumulated in the buffer pool. Specified as number of
bytes ('<int>[bB]?'), megabytes ('<float>[mM]'), gigabytes
('<float>[gG]'), or percentage of the buffer pool limit ('<int>%').
Defaults to bytes if no unit is given..) type: string default: "10%"
This helps prevent Impala accumulating excessive amounts of clean pages,
which can cause some problems in practice:
* The OS buffer cache is squeezed, reducing I/O performance from HDFS
and potentially reducing the ability of the OS to absorb writes from
Impala without blocking.
* Impala process memory consumption can expand more than users or admins
might expect. E.g. if one query is running with a mem_limit of 1GB,
many people will be surprised if the process inflates to the full
process limit of 100GB. Impala doesn't provide any guarantees except
from staying within the process mem_limit, but this could be a
surprising divergence from past behaviour.
Observability:
A new metric buffer-pool.clean-pages-limit is added.
Testing:
Added a backend test to directly test that clean pages are evicted.
Ran in a loop to flush out any flakiness.
Ran exhaustive tests.
Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Reviewed-on: http://gerrit.cloudera.org:8080/7653
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/common/global-flags.cc
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/test-env.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 243 insertions(+), 84 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Tim Armstrong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1092/
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 7: Code-Review+2
A different test that is only built under ASAN failed.
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 4: Code-Review+2
carry
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 3:
(1 comment)
expanded the unit test to test different limits on clean pages.
http://gerrit.cloudera.org:8080/#/c/7653/2/be/src/runtime/bufferpool/buffer-allocator.h
File be/src/runtime/bufferpool/buffer-allocator.h:
PS2, Line 225:
> of?
Done
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1087/
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1087/
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/7653
to look at the new patch set (#7).
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
IMPALA-5677: limit clean page memory consumption
Adds the following flag:
-buffer_pool_clean_pages_limit ((Advanced) Limit on bytes of clean spilled
pages that will be accumulated in the buffer pool. Specified as number of
bytes ('<int>[bB]?'), megabytes ('<float>[mM]'), gigabytes
('<float>[gG]'), or percentage of the buffer pool limit ('<int>%').
Defaults to bytes if no unit is given..) type: string default: "10%"
This helps prevent Impala accumulating excessive amounts of clean pages,
which can cause some problems in practice:
* The OS buffer cache is squeezed, reducing I/O performance from HDFS
and potentially reducing the ability of the OS to absorb writes from
Impala without blocking.
* Impala process memory consumption can expand more than users or admins
might expect. E.g. if one query is running with a mem_limit of 1GB,
many people will be surprised if the process inflates to the full
process limit of 100GB. Impala doesn't provide any guarantees except
from staying within the process mem_limit, but this could be a
surprising divergence from past behaviour.
Observability:
A new metric buffer-pool.clean-pages-limit is added.
Testing:
Added a backend test to directly test that clean pages are evicted.
Ran in a loop to flush out any flakiness.
Ran exhaustive tests.
Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
---
M be/src/common/global-flags.cc
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/test-env.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 243 insertions(+), 84 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/7653/7
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has uploaded a new patch set (#2).
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
IMPALA-5677: limit clean page memory consumption
Adds the following flag:
-buffer_pool_clean_pages_limit ((Advanced) Limit on bytes of clean spilled
pages that will be accumulated in the buffer pool. Specified as number of
bytes ('<int>[bB]?'), megabytes ('<float>[mM]'), gigabytes
('<float>[gG]'), or percentage of the buffer pool limit ('<int>%').
Defaults to bytes if no unit is given..) type: string default: "10%"
This helps prevent Impala accumulating excessive amounts of clean pages,
which can cause some problems in practice:
* The OS buffer cache is squeezed, reducing I/O performance from HDFS
and potentially reducing the ability of the OS to absorb writes from
Impala without blocking.
* Impala process memory consumption can expand more than users or admins
might expect. E.g. if one query is running with a mem_limit of 1GB,
many people will be surprised if the process inflates to the full
process limit of 100GB. Impala doesn't provide any guarantees except
from staying within the process mem_limit, but this could be a
surprising divergence from past behaviour.
Observability:
A new metric buffer-pool.clean-pages-limit is added.
Testing:
Added a backend test to directly test that clean pages are evicted.
Ran in a loop to flush out any flakiness.
Ran exhaustive tests.
Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
---
M be/src/common/global-flags.cc
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/test-env.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 235 insertions(+), 82 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/7653/2
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1089/
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 6: Code-Review+2
Rebase caused a compilation error in unit test
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
Patch Set 3: Code-Review+2
(1 comment)
Looks good, please decide whether the explicit edge case test is warranted.
http://gerrit.cloudera.org:8080/#/c/7653/2/be/src/runtime/bufferpool/buffer-allocator.h
File be/src/runtime/bufferpool/buffer-allocator.h:
PS2, Line 225:
of?
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5677: limit clean page memory consumption
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/7653
to look at the new patch set (#4).
Change subject: IMPALA-5677: limit clean page memory consumption
......................................................................
IMPALA-5677: limit clean page memory consumption
Adds the following flag:
-buffer_pool_clean_pages_limit ((Advanced) Limit on bytes of clean spilled
pages that will be accumulated in the buffer pool. Specified as number of
bytes ('<int>[bB]?'), megabytes ('<float>[mM]'), gigabytes
('<float>[gG]'), or percentage of the buffer pool limit ('<int>%').
Defaults to bytes if no unit is given..) type: string default: "10%"
This helps prevent Impala accumulating excessive amounts of clean pages,
which can cause some problems in practice:
* The OS buffer cache is squeezed, reducing I/O performance from HDFS
and potentially reducing the ability of the OS to absorb writes from
Impala without blocking.
* Impala process memory consumption can expand more than users or admins
might expect. E.g. if one query is running with a mem_limit of 1GB,
many people will be surprised if the process inflates to the full
process limit of 100GB. Impala doesn't provide any guarantees except
from staying within the process mem_limit, but this could be a
surprising divergence from past behaviour.
Observability:
A new metric buffer-pool.clean-pages-limit is added.
Testing:
Added a backend test to directly test that clean pages are evicted.
Ran in a loop to flush out any flakiness.
Ran exhaustive tests.
Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
---
M be/src/common/global-flags.cc
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/suballocator-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/test-env.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/metrics.json
14 files changed, 240 insertions(+), 82 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/7653/4
--
To view, visit http://gerrit.cloudera.org:8080/7653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib6b687ab4bdddf07d9d6c997ff814aa3976042f9
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>