You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2022/08/19 23:37:59 UTC
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18872
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
IMPALA-11514: Workaround s3 connection timeout issues
When running on s3, dataload is failing with errors
like "Timeout waiting for connection from pool". The
underlying issue is a subtle issue in the async draining
codepath (HADOOP-18410). As a temporary workaround, this
adds fs.s3a.input.async.drain.threshold=512G to core-site.xml.
This disables the async drain codepath.
Testing:
- An s3 job passed with this setting
Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/18872/1
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/11195/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Aug 2022 23:58:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8477/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Aug 2022 04:55:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Aug 2022 09:42:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
IMPALA-11514: Workaround s3 connection timeout issues
When running on s3, dataload is failing with errors
like "Timeout waiting for connection from pool". The
underlying issue is a subtle issue in the async draining
codepath (HADOOP-18410). As a temporary workaround, this
adds fs.s3a.input.async.drain.threshold=512G to core-site.xml.
This disables the async drain codepath.
Testing:
- An s3 job passed with this setting
Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Reviewed-on: http://gerrit.cloudera.org:8080/18872
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
---
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py
1 file changed, 3 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Wenzhe Zhou: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
Patch Set 1: Code-Review+1
The change looks good. Are we going to keep this workaround after HADOOP-18410 is fixed?
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Aug 2022 02:38:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
Patch Set 1:
> The change looks good. Are we going to keep this workaround after
> HADOOP-18410 is fixed?
I think we would revert this once HADOOP-18410 is fixed. This workaround is more about getting our tests running, and we don't want to keep this around if it isn't needed. There will be some performance questions, but that seems like those can be handled separately.
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Aug 2022 15:38:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11514: Workaround s3 connection timeout issues
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18872 )
Change subject: IMPALA-11514: Workaround s3 connection timeout issues
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18872
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08d03eb653fdcb6955340519b0cf5ba97b10d590
Gerrit-Change-Number: 18872
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Aug 2022 23:59:31 +0000
Gerrit-HasComments: No