You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org> on 2022/06/16 17:16:42 UTC
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Laszlo Gaal (Cloudera) has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18630
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
IMPALA-10316: Increase Yarn minimum container size for dataload
This is an attempt to get rod of IMPALA-10669 and friends, crashing Tez
containers during the loading of nested ORC data.
It sets the minimum size and the sizing increment of Yarn containers
to 2GB, hoping to avoid too small Yarn containers that sometimes fail
during this step.
Tested by running the dataload phase successfully on the following
platform combinations:
- Ubuntu 16.04, m6i.8xlarge (128 GB RAM, Docker)
- Ubuntu 16.04, m5.12xlarge (192 GB RAM, Docker)
- Centos 7.4, m5.4xlarge (64 GB RAM)
- Centos 7.4, r5.4xlarge (128 GB RAM)
- Ubuntu 16.04, m6i.4xlarge (64 GB RAM)
Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
1 file changed, 4 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/18630/1
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 1
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 4: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Jun 2022 16:15:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10788/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 1
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 17:36:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal (Cloudera) has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
IMPALA-10316: Increase Yarn minimum container size for dataload
This is an attempt to get rod of IMPALA-10669 and friends, crashing Tez
containers during the loading of nested ORC data.
The usual error message logged for these failures is:
Container [pid=11530,containerID=container_1618776748992_0039_01_000003]
is running 2785280B beyond the 'PHYSICAL' memory limit.
Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB
virtual memory used. Killing container.
https://stackoverflow.com/a/43827548/143681 explains that the tunable
setting 'yarn.scheduler.minimum-allocation-mb' in yarn-site.xml sets
both the minimum memory size and the memory size increment for Yarn
containers
This patch is an attempt to work around the failure by forcibly setting
a minimum size for the Yarn containers used in dataload that is
significantly larger than the 1 GB size reported in the failure messages.
Tested by running the dataload phase successfully on the following
platform combinations:
- Ubuntu 16.04, m6i.8xlarge (128 GB RAM, Docker)
- Ubuntu 16.04, m5.12xlarge (192 GB RAM, Docker)
- Centos 7.4, m5.4xlarge (64 GB RAM)
- Centos 7.4, r5.4xlarge (128 GB RAM)
- Ubuntu 16.04, m6i.4xlarge (64 GB RAM)
Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Reviewed-on: http://gerrit.cloudera.org:8080/18630
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Laszlo Gaal (Cloudera) <la...@cloudera.com>
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
1 file changed, 4 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Verified
Michael Smith: Looks good to me, but someone else must approve
Laszlo Gaal (Cloudera): Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 5
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal (Cloudera) has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 4: Code-Review+1
Carry +1 from Michael Smith
Edited the commit message to remove the dashed lines. They confused the commit-message hook,making it add another Gerrit ChangeId before them.
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 21:17:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Jun 2022 01:57:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 1:
This does increase the memory allocated for TezChild containers started during ORC load. So I agree it should also address the problem by increasing the memory available for non-heap use (from 205MB to 410MB).
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 1
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 17:26:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18630
to look at the new patch set (#2).
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
IMPALA-10316: Increase Yarn minimum container size for dataload
This is an attempt to get rod of IMPALA-10669 and friends, crashing Tez
containers during the loading of nested ORC data.
The usual error message logged for these failures is:
Change-Id: Id8ebab8ebb07357744115298c163351c77135c08
--------------------------------------------------
Container [pid=11530,containerID=container_1618776748992_0039_01_000003]
is running 2785280B beyond the 'PHYSICAL' memory limit.
Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB
virtual memory used. Killing container.
--------------------------------------------------
https://stackoverflow.com/a/43827548/143681 explains that the tunable
setting 'yarn.scheduler.minimum-allocation-mb' in yarn-site.xml sets
both the minimum memory size and the memory size increment for Yarn
containers
This patch is an attempt to work around the failure by forcibly setting
a minimum size for the Yarn containers used in dataload that is
significantly larger than the 1 GB size reported in the failure messages.
Tested by running the dataload phase successfully on the following
platform combinations:
- Ubuntu 16.04, m6i.8xlarge (128 GB RAM, Docker)
- Ubuntu 16.04, m5.12xlarge (192 GB RAM, Docker)
- Centos 7.4, m5.4xlarge (64 GB RAM)
- Centos 7.4, r5.4xlarge (128 GB RAM)
- Ubuntu 16.04, m6i.4xlarge (64 GB RAM)
Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
1 file changed, 4 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/18630/2
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 2
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal (Cloudera) has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 2: Code-Review+1
Carry +1 from Michael Smith
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 2
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 21:11:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 1:
I'll evaluate this similar to what I've done for https://gerrit.cloudera.org/c/18553/ and https://gerrit.cloudera.org/c/18612/. Quanlong has advocated for a more targeted change (only for the ORC loading).
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 1
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 17:23:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 1
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 17:26:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal (Cloudera) has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Jun 2022 18:44:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18630
to look at the new patch set (#3).
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
IMPALA-10316: Increase Yarn minimum container size for dataload
This is an attempt to get rod of IMPALA-10669 and friends, crashing Tez
containers during the loading of nested ORC data.
The usual error message logged for these failures is:
Change-Id: Ic14bfd402c70e93204dac67579b984f44cb41091
--------------------------------------------------
Container [pid=11530,containerID=container_1618776748992_0039_01_000003]
is running 2785280B beyond the 'PHYSICAL' memory limit.
Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB
virtual memory used. Killing container.
--------------------------------------------------
https://stackoverflow.com/a/43827548/143681 explains that the tunable
setting 'yarn.scheduler.minimum-allocation-mb' in yarn-site.xml sets
both the minimum memory size and the memory size increment for Yarn
containers
This patch is an attempt to work around the failure by forcibly setting
a minimum size for the Yarn containers used in dataload that is
significantly larger than the 1 GB size reported in the failure messages.
Tested by running the dataload phase successfully on the following
platform combinations:
- Ubuntu 16.04, m6i.8xlarge (128 GB RAM, Docker)
- Ubuntu 16.04, m5.12xlarge (192 GB RAM, Docker)
- Centos 7.4, m5.4xlarge (64 GB RAM)
- Centos 7.4, r5.4xlarge (128 GB RAM)
- Ubuntu 16.04, m6i.4xlarge (64 GB RAM)
Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
1 file changed, 4 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/18630/3
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 3
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Laszlo Gaal (Cloudera) (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18630
to look at the new patch set (#4).
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
IMPALA-10316: Increase Yarn minimum container size for dataload
This is an attempt to get rod of IMPALA-10669 and friends, crashing Tez
containers during the loading of nested ORC data.
The usual error message logged for these failures is:
Container [pid=11530,containerID=container_1618776748992_0039_01_000003]
is running 2785280B beyond the 'PHYSICAL' memory limit.
Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB
virtual memory used. Killing container.
https://stackoverflow.com/a/43827548/143681 explains that the tunable
setting 'yarn.scheduler.minimum-allocation-mb' in yarn-site.xml sets
both the minimum memory size and the memory size increment for Yarn
containers
This patch is an attempt to work around the failure by forcibly setting
a minimum size for the Yarn containers used in dataload that is
significantly larger than the 1 GB size reported in the failure messages.
Tested by running the dataload phase successfully on the following
platform combinations:
- Ubuntu 16.04, m6i.8xlarge (128 GB RAM, Docker)
- Ubuntu 16.04, m5.12xlarge (192 GB RAM, Docker)
- Centos 7.4, m5.4xlarge (64 GB RAM)
- Centos 7.4, r5.4xlarge (128 GB RAM)
- Ubuntu 16.04, m6i.4xlarge (64 GB RAM)
Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
1 file changed, 4 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/18630/4
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-10316: Increase Yarn minimum container size for dataload
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18630 )
Change subject: IMPALA-10316: Increase Yarn minimum container size for dataload
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8242/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Gerrit-Change-Number: 18630
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal (Cloudera) <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jun 2022 21:19:18 +0000
Gerrit-HasComments: No