You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org> on 2022/08/15 18:29:54 UTC

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18851


Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................

IMPALA-11498: Change port range of TEZ's web UI server

After TEZ-4347, by default TEZ would attempt to start a web UI server
before opening a session. The default port range for the server
specified in TezConfiguration.java (in the TEZ repository) is
"50000-50050", which does not seem to be a good choice in Impala's
testing environment in that there are always some other client programs
holding those ports when TEZ attempts to start its web UI server. As a
result, TEZ could not bind a port in the port range to start its web UI
server, resulting in TEZ session not being created, which in turn fail
some end-to-end test relying on TEZ, e.g., test_json_file_unsupported.

This patch explicitly sets the port range to "32000-32100" to avoid port
collision based on the parameter of 'ip_local_port_range', which is
"32768-60999" on the machine where Impala's tests are run.

Testing:
 - Verified in 2 Jenkins runs of exhaustive RELEASE build that we do not
   see test_json_file_unsupported fail again after this patch.

Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 1 insertion(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/18851/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................

IMPALA-11498: Change port range of TEZ's web UI server

After TEZ-4347, by default TEZ would attempt to start a web UI server
before opening a session. The default port range for the server
specified in TezConfiguration.java (in the TEZ repository) is
"50000-50050", which does not seem to be a good choice in Impala's
testing environment in that there are always some other client programs
holding those ports when TEZ attempts to start its web UI server. As a
result, TEZ could not bind a port in the port range to start its web UI
server, resulting in TEZ session not being created, which in turn failed
some end-to-end test relying on TEZ, e.g., test_json_file_unsupported.

This patch explicitly sets the port range to "32000-32100" to avoid port
collision based on the parameter of 'ip_local_port_range', which is
"32768-60999" on the machine where Impala's tests are run.

Testing:
 - Verified in 2 Jenkins runs of exhaustive RELEASE build that we do not
   see test_json_file_unsupported fail again after this patch.

Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/18851/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Aug 2022 05:29:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8453/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Aug 2022 00:39:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11157/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 19:01:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11156/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 18:50:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 2: Code-Review+2

This makes sense to me. Thanks for putting this together!


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Aug 2022 00:38:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................

IMPALA-11498: Change port range of TEZ's web UI server

After TEZ-4347, by default TEZ would attempt to start a web UI server
before opening a session. The default port range for the server
specified in TezConfiguration.java (in the TEZ repository) is
"50000-50050", which does not seem to be a good choice in Impala's
testing environment in that there are always some other client programs
holding those ports when TEZ attempts to start its web UI server. As a
result, TEZ could not bind a port in the port range to start its web UI
server, resulting in TEZ session not being created, which in turn failed
some end-to-end test relying on TEZ, e.g., test_json_file_unsupported.

This patch explicitly sets the port range to "32000-32100" to avoid port
collision based on the parameter of 'ip_local_port_range', which is
"32768-60999" on the machine where Impala's tests are run.

Testing:
 - Verified in 2 Jenkins runs of exhaustive RELEASE build that we do not
   see test_json_file_unsupported fail again after this patch.

Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Reviewed-on: http://gerrit.cloudera.org:8080/18851
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 4
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 1:

Hi all, please let me know if you have any comment on the patch. Thank you very much for the help!


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 18:35:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11498: Change port range of TEZ's web UI server

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18851 )

Change subject: IMPALA-11498: Change port range of TEZ's web UI server
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d067953f943a762ad50b0561570e96f0dc39791
Gerrit-Change-Number: 18851
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: John Sherman <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Aug 2022 00:39:00 +0000
Gerrit-HasComments: No