You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2024/08/09 22:21:40 UTC
[Impala-ASF-CR] WIP: quicker cluster startup
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21656
Change subject: WIP: quicker cluster startup
......................................................................
WIP: quicker cluster startup
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
---
M tests/common/impala_cluster.py
M tests/common/impala_service.py
2 files changed, 19 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/21656/1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21656/4/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/21656/4/tests/common/impala_cluster.py@625
PS4, Line 625: #
flake8: E265 block comment should start with '# '
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 07:16:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 6: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py@622
PS5, Line 622: coordinat
> nit: typo
Done
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py@625
PS5, Line 625: le time.time() - start_time < CLUSTER_WAIT_TI
> I think it should (especially if regular core-mode tests use that port as w
I added waiting for hs2-http port. It only checks whether the port is open and doesn't try to establish a thrift connection. Note sure how much the actual connections add - Impala should only open the porn when it is ready to accept hs2 connections.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Aug 2024 15:26:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21656
to look at the new patch set (#3).
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
IMPALA-11729: Speed up start-impala-cluster.py
The change reduces cluster startup time by 1-2 seconds. This also
makes custom cluster tests a bit quicker.
Most of the improvement is caused by removing unneeded sleep from
wait_for_catalog() - it also slept after successful connections,
while when the first coordinator is up, it is likely that all
others are also up, meaning 3*0.5s extra sleep in the dev cluster.
Other changes:
- wait_for_catalog is cleaned up and renamed to
wait_for_coordinator_services
- decreased some sleep intervals
- removed some non-informative logging
- wait for hs2/beeswax/webui ports to be open before trying
to actually connect to them to avoid extra logging from
failed Thrift/http connections
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
---
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
3 files changed, 79 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/21656/3
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10856/
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 12:25:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16660/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 19:14:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16661/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 19:17:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21656
to look at the new patch set (#6).
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
IMPALA-11729: Speed up start-impala-cluster.py
The change reduces cluster startup time by 1-2 seconds. This also
makes custom cluster tests a bit quicker.
Most of the improvement is caused by removing unneeded sleep from
wait_for_catalog() - it also slept after successful connections,
while when the first coordinator is up, it is likely that all
others are also up, meaning 3*0.5s extra sleep in the dev cluster.
Other changes:
- wait_for_catalog is cleaned up and renamed to
wait_for_coordinator_services
- also wait for hs2_http port to be open
- decreased some sleep intervals
- removed some non-informative logging
- wait for hs2/beeswax/webui ports to be open before trying
to actually connect to them to avoid extra logging from
failed Thrift/http connections
- reordered startup to first wait for coordinators to be up
then wait for num_known_live_backends in each impalad - this
reflects better what the cluster actually waits for (1st catalog
update before starting coordinator services)
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
---
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
3 files changed, 88 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/21656/6
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Aug 2024 17:17:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
IMPALA-11729: Speed up start-impala-cluster.py
The change reduces cluster startup time by 1-2 seconds. This also
makes custom cluster tests a bit quicker.
Most of the improvement is caused by removing unneeded sleep from
wait_for_catalog() - it also slept after successful connections,
while when the first coordinator is up, it is likely that all
others are also up, meaning 3*0.5s extra sleep in the dev cluster.
Other changes:
- wait_for_catalog is cleaned up and renamed to
wait_for_coordinator_services
- also wait for hs2_http port to be open
- decreased some sleep intervals
- removed some non-informative logging
- wait for hs2/beeswax/webui ports to be open before trying
to actually connect to them to avoid extra logging from
failed Thrift/http connections
- reordered startup to first wait for coordinators to be up
then wait for num_known_live_backends in each impalad - this
reflects better what the cluster actually waits for (1st catalog
update before starting coordinator services)
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Reviewed-on: http://gerrit.cloudera.org:8080/21656
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Laszlo Gaal <la...@cloudera.com>
---
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
3 files changed, 88 insertions(+), 39 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Michael Smith: Looks good to me, but someone else must approve
Laszlo Gaal: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10854/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 18:47:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10854/
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 23:48:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: quicker cluster startup
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: WIP: quicker cluster startup
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 10 Aug 2024 03:16:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16669/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 07:47:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5: Code-Review+1
(2 comments)
LGTM, just a typo.
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py@622
PS5, Line 622: coordinor
nit: typo
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py@625
PS5, Line 625: TODO: shouldn't this also wait for hs2-http?
I think it should (especially if regular core-mode tests use that port as well), but I'm OK with that being added in a follow-up patch.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Aug 2024 13:47:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sun, 06 Oct 2024 00:52:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 7: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py@622
PS5, Line 622: coordinat
> Done
Ack
http://gerrit.cloudera.org:8080/#/c/21656/5/tests/common/impala_cluster.py@625
PS5, Line 625: le time.time() - start_time < CLUSTER_WAIT_TI
> I added waiting for hs2-http port. It only checks whether the port is open
Done
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Aug 2024 18:24:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21656
to look at the new patch set (#4).
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
IMPALA-11729: Speed up start-impala-cluster.py
The change reduces cluster startup time by 1-2 seconds. This also
makes custom cluster tests a bit quicker.
Most of the improvement is caused by removing unneeded sleep from
wait_for_catalog() - it also slept after successful connections,
while when the first coordinator is up, it is likely that all
others are also up, meaning 3*0.5s extra sleep in the dev cluster.
Other changes:
- wait_for_catalog is cleaned up and renamed to
wait_for_coordinator_services
- decreased some sleep intervals
- removed some non-informative logging
- wait for hs2/beeswax/webui ports to be open before trying
to actually connect to them to avoid extra logging from
failed Thrift/http connections
- reordered startup to first wait for coordinators to be up
then wait for num_known_live_backends in each impalad - this
reflects better what the cluster actually waits for (1st catalog
update before starting coordinator services)
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
---
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
3 files changed, 83 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/21656/4
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11000/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Oct 2024 19:50:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: quicker cluster startup
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: WIP: quicker cluster startup
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_cluster.py@44
PS1, Line 44: #logging.basicConfig(level=logging.INFO,
flake8: E265 block comment should start with '# '
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_cluster.py@47
PS1, Line 47: #)
flake8: E265 block comment should start with '# '
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_cluster.py@234
PS1, Line 234: =
flake8: E225 missing whitespace around operator
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_cluster.py@234
PS1, Line 234:
flake8: E261 at least two spaces before inline comment
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_service.py
File tests/common/impala_service.py:
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_service.py@266
PS1, Line 266: #
flake8: E265 block comment should start with '# '
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_service.py@375
PS1, Line 375:
flake8: E222 multiple spaces after operator
http://gerrit.cloudera.org:8080/#/c/21656/1/tests/common/impala_service.py@375
PS1, Line 375:
flake8: E261 at least two spaces before inline comment
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2024 22:22:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/16810/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Aug 2024 15:55:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21656/2/tests/common/impala_service.py
File tests/common/impala_service.py:
http://gerrit.cloudera.org:8080/#/c/21656/2/tests/common/impala_service.py@434
PS2, Line 434: e
flake8: F841 local variable 'e' is assigned to but never used
http://gerrit.cloudera.org:8080/#/c/21656/2/tests/common/impala_service.py@458
PS2, Line 458: e
flake8: F821 undefined name 'e'
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 18:44:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 23:33:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21656
to look at the new patch set (#5).
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
IMPALA-11729: Speed up start-impala-cluster.py
The change reduces cluster startup time by 1-2 seconds. This also
makes custom cluster tests a bit quicker.
Most of the improvement is caused by removing unneeded sleep from
wait_for_catalog() - it also slept after successful connections,
while when the first coordinator is up, it is likely that all
others are also up, meaning 3*0.5s extra sleep in the dev cluster.
Other changes:
- wait_for_catalog is cleaned up and renamed to
wait_for_coordinator_services
- decreased some sleep intervals
- removed some non-informative logging
- wait for hs2/beeswax/webui ports to be open before trying
to actually connect to them to avoid extra logging from
failed Thrift/http connections
- reordered startup to first wait for coordinators to be up
then wait for num_known_live_backends in each impalad - this
reflects better what the cluster actually waits for (1st catalog
update before starting coordinator services)
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
---
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
3 files changed, 83 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/21656/5
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] WIP: quicker cluster startup
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: WIP: quicker cluster startup
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10852/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2024 22:22:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: quicker cluster startup
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: WIP: quicker cluster startup
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16651/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Aug 2024 22:54:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16670/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 07:47:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10856/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 07:21:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 6:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python2 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with virtualenv installed.
http://gerrit.cloudera.org:8080/#/c/21656/6/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/21656/6/tests/common/impala_cluster.py@622
PS6, Line 622: +
flake8: W504 line break after binary operator
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Aug 2024 15:26:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has removed a vote on this change.
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Oct 2024 16:11:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 9: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Oct 2024 15:16:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5:
PS 3/4 only contain small change in logging/comments/commit message
the jenkins run failed due to custom_cluster.test_statestored_ha.TestStatestoredHA.test_statestored_auto_failover - I was unable to reproduce the failure after running the test >= 100 times, so my assumption is that this is a very rare flaky test
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 07:20:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5:
The latest jenkins run hit test_exchange_mem_usage_scaling, which is a known flaky test.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 12:47:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 13 Aug 2024 16:46:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21656
to look at the new patch set (#2).
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
IMPALA-11729: Speed up start-impala-cluster.py
The change reduces cluster startup time by 1-2 seconds. This also
makes custom cluster tests a bit quicker.
Most of the improvement is caused by removing unneeded sleep from
wait_for_catalog() - it also slept after successful connections,
while when the first coordinator is up, it is likely that all
others are also up, meaning 3*0.5s extra sleep in the dev cluster.
Other changes:
- wait_for_catalog is cleaned up and renamed to
wait_for_coordinator_services
- decreased some sleep intervals
- removed some non-informative logging
- wait for hs2/beeswax/webui ports to be open before trying
to actually connect to them to avoid extra logging from
failed Thrift/http connections
Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
---
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
3 files changed, 78 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/21656/2
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10914/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Aug 2024 12:27:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 8: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Aug 2024 16:24:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11729: Speed up start-impala-cluster.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21656 )
Change subject: IMPALA-11729: Speed up start-impala-cluster.py
......................................................................
Patch Set 7:
> Patch Set 7: Patch Set 6 was rebased
Base was old enough it ran into IMPALA-12331.
--
To view, visit http://gerrit.cloudera.org:8080/21656
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4dd8c2bc7056443373ceb256a03ce562fea38a0
Gerrit-Change-Number: 21656
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Aug 2024 18:22:50 +0000
Gerrit-HasComments: No