You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "wangning (Code Review)" <ge...@cloudera.org> on 2020/04/28 03:46:09 UTC
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
wangning has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15830
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
[fs] fixed file num low live ratio metadata compaction policy when
startup
During tserver startup, it loads all low live ratio metadata into memory,
and this is not under controll of --memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, num of metadata files increases
and may result in OOM in next time startup.
Passing the --log_container_live_metadata_before_compact_ratio would
help limit the memory consumptio when startup.
To limit the memory consumption of livemetadata compaction, we always
use --log_container_live_metadata_before_compact_ratio would help. But it is
not flexible cause we have no idea about the distribution of live ratio
of blocks.
Limit the num of metadata files to be compacted could help avoid OOM.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 13 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/1
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 1
Gerrit-Owner: wangning <19...@gmail.com>
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15830
to look at the new patch set (#4).
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
[fs] fixed file num low live ratio metadata compaction policy when
startup
During tserver startup, it loads all low live ratio metadata into memory,
and this is not under controll of --memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, num of metadata files increases
and may result in OOM in next time startup. Sometimes the memory
consumption for startup may excess physical memory.
To limit the memory consumption of livemetadata compaction, we always
use --log_container_live_metadata_before_compact_ratio. But it is
not flexible cause we have no idea about the distribution of live ratio
of blocks.
Limit the num of metadata files to be compacted could help avoid OOM
during startup.
E.g. In a node with 32G physical memory, we found about 90GiB metatatada
files, the num of metadata files is 200k. It would be very good to
compact them all when startup. But it could take about 90GiB * 60% =
54GiB memory when starting(Not precise, as we observed is consume 60% of
metadata bytes in disk), obviously it excess the physical memory. So it keeps
OOM and cdh-agent keeps restart it. By passing the
--max_num_low_live_ratio_metadata_file_to_compact to limit the num of
low live ratio files could be selected to be compact, could make it
success to startup.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 14 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/4
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 4
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
wangning has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@18
PS2, Line 18: nsumption
> nit: consumption
Done
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@20
PS2, Line 20: ion of live
> nit: remove.
Done
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@24
PS2, Line 24: during startup.
> Can you explain more about how this flag (which is to limit the number of m
For sure it would be very good to make those files compacted. But it is no possible for nodes without sufficient memory to startup. If sometimes it killed by OOM killer , it may keeps fail forever without intervention of Ops.
The metadata files inflate over physical memory is very common to us, I am in private deployment company which offer service to other company. The management right doesn't belong to us until some module failed. Even if we monitored the bad situation of compaction of kudu tserver, it is not possible to restart and adjust some parameter without customer's permission.
How the flag could help avoiding OOM during startup is attached in CL.
We some times cut in half of num of metadata files to start once, after some compactions and remove the flag and restart it again. With 1.5 times expectation of start time but could help the node restart even in not very good hardware.
http://gerrit.cloudera.org:8080/#/c/15830/2/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15830/2/src/kudu/fs/log_block_manager.cc@2631
PS2, Line 2631: FLAGS_max_num_low_live_ratio_metadata_file_to_compact >
: l
> nit: alignment.
Done
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 3
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Apr 2020 00:59:26 +0000
Gerrit-HasComments: Yes
[kudu-CR] [fs] limit memory could consume compacting low live ratio blocks during startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15830
to look at the new patch set (#6).
Change subject: [fs] limit memory could consume compacting low live ratio blocks during startup
......................................................................
[fs] limit memory could consume compacting low live ratio blocks during startup
During tserver startup, it loads all blocks of low live ratio metadata into
memory for compacting, and the memory behave is not under control of
FLAGS_memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, size of all metadata files increases
and may result in OOM in next startup. Sometimes the memory consumption for
startup may exceed physical memory.
E.g. In a node with 32G physical memory, we found about 90GiB metatatada
files, to load all the low live ratio blocks into memory may take over
50g memory. It's far exceeds the memory limit of kudu and even physical
limit. Once it failed due to some reason or restart it manually, it may
never make it startup.
Here introduce FlAGS_max_limit_hard_bytes_low_live_ratio_blocks_compacting to
limit the memory may take by compacting low live blocks.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 16 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/6
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 6
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15830
to look at the new patch set (#2).
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
[fs] fixed file num low live ratio metadata compaction policy when
startup
During tserver startup, it loads all low live ratio metadata into memory,
and this is not under controll of --memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, num of metadata files increases
and may result in OOM in next time startup.
Passing the --log_container_live_metadata_before_compact_ratio would
help limit the memory consumptio when startup.
To limit the memory consumption of livemetadata compaction, we always
use --log_container_live_metadata_before_compact_ratio would help. But it is
not flexible cause we have no idea about the distribution of live ratio
of blocks.
Limit the num of metadata files to be compacted could help avoid OOM.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 14 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/2
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 2
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [fs] limit memory could consumpt by low live ratio blocks compacting during startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15830
to look at the new patch set (#5).
Change subject: [fs] limit memory could consumpt by low live ratio blocks compacting during startup
......................................................................
[fs] limit memory could consumpt by low live ratio blocks compacting during startup
During tserver startup, it loads all blocks of low live ratio metadata into
memory for compacting, and the memory behave is not under control of
FLAGS_memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, size of all metadata files increases
and may result in OOM in next startup. Sometimes the memory consumption for
startup may exceed physical memory.
E.g. In a node with 32G physical memory, we found about 90GiB metatatada
files, to load all the low live ratio blocks into memory may take over
50g memory. It's far exceeds the memory limit of kudu and even physical
limit. Once it failed due to some reason or restart it manually, it may
never make it startup.
Here introduce FlAGS_max_limit_hard_bytes_low_live_ratio_blocks_compacting to
limit the memory may take by compacting low live blocks.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 16 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/5
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 5
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@7
PS4, Line 7: [fs] fixed file num low live ratio metadata compaction policy when
: startup
> nit: keep the title on a single line
Also I think this would be a clearer:
"[fs] limit to the number of metadata file compactions done at startup"
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@2668
PS4, Line 2668: low_live_ratio_metadata_to_compact_num_.Increment();
> Using the number of metadata files seems kind of odd, given it requires som
Looking at the definition of BlockRecordPB, it seems to have the following members:
::google::protobuf::internal::InternalMetadataWithArena _internal_metadata_;
::google::protobuf::internal::HasBits<1> _has_bits_;
mutable int _cached_size_;
::kudu::BlockIdPB* block_id_;
::google::protobuf::uint64 timestamp_us_;
::google::protobuf::int64 offset_;
::google::protobuf::int64 length_;
int op_type_;
Most of these seem like trivial types, so we might be able to use `sizeof(BlockRecordPB) + sizeof(BlockIdPB)` as the individual size of a record on the heap (i.e. per element in the vector).
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 4
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
Gerrit-Comment-Date: Thu, 14 May 2020 05:56:33 +0000
Gerrit-HasComments: Yes
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
Patch Set 4:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@7
PS4, Line 7: [fs] fixed file num low live ratio metadata compaction policy when
: startup
nit: keep the title on a single line
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@10
PS4, Line 10: it loads all low live ratio metadata into memory,
nit: I wasn't sure what "low live ratio metadata" referred to. Could you also add a small description here, e.g.
"it loads all low live ratio metadata into memory (i.e. metadata for containers that contain a low number of live blocks), and this is..."
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@11
PS4, Line 11: controll
nit: "control" with one 'l'
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@16
PS4, Line 16: excess
nit: "exceed"
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@18
PS4, Line 18: livemetadata
nit: add a space: "live metadata"
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@29
PS4, Line 29: rting(Not
nit: add a space in between "starting (", and "Not" should be lower cased.
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@30
PS4, Line 30: excess
nit: "exceeds"
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@119
PS4, Line 119: DEFINE_int32(max_num_low_live_ratio_metadata_file_to_compact, -1,
nit: can you put this up by log_container_live_metadata_before_compact_ratio so the compaction-related flags are adjacent to one another?
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@120
PS4, Line 120:
nit: spell out "number"
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@2668
PS4, Line 2668: low_live_ratio_metadata_to_compact_num_.Increment();
Using the number of metadata files seems kind of odd, given it requires some experimentation or knowledge beforehand about what is being stored in memory to use it effectively.
Would it be possible to measure size via BlockRecordPB::ByteSizeLong() and limit the number of bytes we're allowed to collect for compaction instead? As far as I can tell, ByteSizeLong() is the post-encoding size, so it may not be accurate. But if we could get the in-memory size of the records, we could apply some bytes limit, which seems more natural.
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 4
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
Gerrit-Comment-Date: Thu, 14 May 2020 03:51:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] [fs] limit memory could consume compacting low live ratio blocks during startup
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] limit memory could consume compacting low live ratio blocks during startup
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15830/6//COMMIT_MSG
Commit Message:
PS6:
Would it be possible to add a test for this? Maybe use CurrentConsumption() to periodically check the memory usage and see that it doesn't exceed some value.
Have you verified that this new code works on a test server?
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@2668
PS4, Line 2668: });
> Done
I looked around the codebase a bit more and it seems like we use this kudu_malloc_usable_size(). I'm not sure it's exactly what we're looking for, but it'd be great if you could experiment a bit to see if it's appropriate to use, vs what we have now with the sizeof().
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 6
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
Gerrit-Comment-Date: Thu, 21 May 2020 22:36:59 +0000
Gerrit-HasComments: Yes
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@18
PS2, Line 18: consumptio
nit: consumption
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@20
PS2, Line 20: would help
nit: remove.
http://gerrit.cloudera.org:8080/#/c/15830/2//COMMIT_MSG@24
PS2, Line 24: Limit the num of metadata files to be compacted could help avoid OOM.
Can you explain more about how this flag (which is to limit the number of metadata files to be compacted) can help avoid OOM? Ideally, wouldn't the more compaction of metadata files (number of metadata files decrease), the less the memory will be used?
http://gerrit.cloudera.org:8080/#/c/15830/2/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15830/2/src/kudu/fs/log_block_manager.cc@2631
PS2, Line 2631: FLAGS_max_num_low_live_ratio_metadata_file_to_compact >
: lo
nit: alignment.
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 2
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 28 Apr 2020 23:08:29 +0000
Gerrit-HasComments: Yes
[kudu-CR] [fs] fixed file num low live ratio metadata compaction policy when startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15830
to look at the new patch set (#3).
Change subject: [fs] fixed file num low live ratio metadata compaction policy when startup
......................................................................
[fs] fixed file num low live ratio metadata compaction policy when
startup
During tserver startup, it loads all low live ratio metadata into memory,
and this is not under controll of --memory_limit_hard_bytes.
To those tservers which didn't compact well due to starvation of
compaction, the metadata files inflated, num of metadata files increases
and may result in OOM in next time startup. Sometimes the memory
consumption for startup may excess physical memory.
To limit the memory consumption of livemetadata compaction, we always
use --log_container_live_metadata_before_compact_ratio. But it is
not flexible cause we have no idea about the distribution of live ratio
of blocks.
Limit the num of metadata files to be compacted could help avoid OOM
during startup.
E.g. In a node with 32G physical memory, we found about 90GiB metatatada
files, the num of metadata files is 200k. It would be very good to
compact them all when startup. But it could take about 90GiB * 60% =
48GiB memory when starting(Not precise, as we observed is consume 60% of
metadata bytes in disk), obviously it excess the physical memory. So it keeps
OOM and cdh-agent keeps restart it. By passing the
--max_num_low_live_ratio_metadata_file_to_compact to limit the num of
low live ratio files could be selected to be compact, could make it
success to startup.
Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 14 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/15830/3
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 3
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [fs] limit memory could consume compacting low live ratio blocks during startup
Posted by "wangning (Code Review)" <ge...@cloudera.org>.
wangning has posted comments on this change. ( http://gerrit.cloudera.org:8080/15830 )
Change subject: [fs] limit memory could consume compacting low live ratio blocks during startup
......................................................................
Patch Set 6:
(8 comments)
Thx for point out. My original purpose was to limit the memory.
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@7
PS4, Line 7: [fs] limit memory could consume compacting low live ratio blocks during startup
:
> Also I think this would be a clearer:
Change the commit to limit the memory may consume.
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@10
PS4, Line 10: nd the memory behave is not under control of
> nit: I wasn't sure what "low live ratio metadata" referred to. Could you al
It refers to all blocks in low ratio blocks. Changed to a clearer title.
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@11
PS4, Line 11: d_bytes.
> nit: "control" with one 'l'
Done
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@16
PS4, Line 16: memory
> nit: "exceed"
Done
http://gerrit.cloudera.org:8080/#/c/15830/4//COMMIT_MSG@30
PS4, Line 30:
> nit: "exceeds"
Done
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@119
PS4, Line 119: "startup time");
> nit: can you put this up by log_container_live_metadata_before_compact_rati
Done
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@120
PS4, Line 120: n
> nit: spell out "number"
Done
http://gerrit.cloudera.org:8080/#/c/15830/4/src/kudu/fs/log_block_manager.cc@2668
PS4, Line 2668: });
> Looking at the definition of BlockRecordPB, it seems to have the following
Done
--
To view, visit http://gerrit.cloudera.org:8080/15830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifad1d6de4e61a1ddcfb35743abd71b57d6418896
Gerrit-Change-Number: 15830
Gerrit-PatchSet: 6
Gerrit-Owner: wangning <19...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: wangning <19...@gmail.com>
Gerrit-Comment-Date: Fri, 15 May 2020 03:23:51 +0000
Gerrit-HasComments: Yes