You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2018/10/16 21:25:38 UTC
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11701
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Two test_udfs.py tests (test_native_functions_race and
test_concurrent_jar_drop_use) spawn dozens of connections to
test Impala behavior under concurrency. These connections
use up frontend service threads and can cause shell tests
to timeout when trying to connect.
This moves both tests to a new TestUdfConcurrency custom
cluster test. The new custom cluster test uses a larger
fe_service_threads value to allow full concurrency. The
tests run serially and cannot impact other tests.
This also reduces the test dimensions for test_native_functions_race
so that it runs one configuration rather than eight.
Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
---
A tests/custom_cluster/test_udf_concurrency.py
M tests/query_test/test_udfs.py
2 files changed, 206 insertions(+), 162 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/11701/1
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 2: Code-Review+2
Fixed flake8 issues. Carrying +2
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Oct 2018 17:31:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3325/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Oct 2018 17:31:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Oct 2018 21:56:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11701
to look at the new patch set (#2).
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Two test_udfs.py tests (test_native_functions_race and
test_concurrent_jar_drop_use) spawn dozens of connections to
test Impala behavior under concurrency. These connections
use up frontend service threads and can cause shell tests
to timeout when trying to connect.
This moves both tests to a new TestUdfConcurrency custom
cluster test. The new custom cluster test uses a larger
fe_service_threads value to allow full concurrency. The
tests run serially and cannot impact other tests.
This also reduces the test dimensions for test_native_functions_race
so that it runs one configuration rather than eight.
Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
---
A tests/custom_cluster/test_udf_concurrency.py
M tests/query_test/test_udfs.py
2 files changed, 206 insertions(+), 162 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/11701/2
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 1:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py
File tests/custom_cluster/test_udf_concurrency.py:
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@23
PS1, Line 23: from subprocess import call, check_call
flake8: F401 'subprocess.call' imported but unused
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@27
PS1, Line 27: from tests.common.test_dimensions import (
flake8: F401 'tests.common.test_dimensions.create_exec_option_dimension_from_dict' imported but unused
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@27
PS1, Line 27: from tests.common.test_dimensions import (
flake8: F401 'tests.common.test_dimensions.create_exec_option_dimension' imported but unused
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@27
PS1, Line 27: from tests.common.test_dimensions import (
flake8: F401 'tests.common.test_dimensions.create_uncompressed_text_dimension' imported but unused
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@31
PS1, Line 31: from tests.common.test_vector import ImpalaTestDimension
flake8: F401 'tests.common.test_vector.ImpalaTestDimension' imported but unused
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@37
PS1, Line 37: class TestUdfConcurrency(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@56
PS1, Line 56: a
flake8: E501 line too long (97 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@84
PS1, Line 84: d
flake8: E306 expected 1 blank line before a nested definition, found 0
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@121
PS1, Line 121: ;
flake8: E703 statement ends with a semicolon
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@174
PS1, Line 174: d
flake8: E306 expected 1 blank line before a nested definition, found 0
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@194
PS1, Line 194: d
flake8: F841 local variable 'drop_threads' is assigned to but never used
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@202
PS1, Line 202: ;
flake8: E703 statement ends with a semicolon
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Oct 2018 23:55:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Two test_udfs.py tests (test_native_functions_race and
test_concurrent_jar_drop_use) spawn dozens of connections to
test Impala behavior under concurrency. These connections
use up frontend service threads and can cause shell tests
to timeout when trying to connect.
This moves both tests to a new TestUdfConcurrency custom
cluster test. The new custom cluster test uses a larger
fe_service_threads value to allow full concurrency. The
tests run serially and cannot impact other tests.
This also reduces the test dimensions for test_native_functions_race
so that it runs one configuration rather than eight.
Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Reviewed-on: http://gerrit.cloudera.org:8080/11701
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
A tests/custom_cluster/test_udf_concurrency.py
M tests/query_test/test_udfs.py
2 files changed, 206 insertions(+), 162 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Oct 2018 21:25:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 1:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py
File tests/custom_cluster/test_udf_concurrency.py:
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@23
PS1, Line 23: from subprocess import call, check_call
> flake8: F401 'subprocess.call' imported but unused
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@27
PS1, Line 27: from tests.common.test_dimensions import (
> flake8: F401 'tests.common.test_dimensions.create_exec_option_dimension_fro
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@27
PS1, Line 27: from tests.common.test_dimensions import (
> flake8: F401 'tests.common.test_dimensions.create_uncompressed_text_dimensi
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@27
PS1, Line 27: from tests.common.test_dimensions import (
> flake8: F401 'tests.common.test_dimensions.create_exec_option_dimension' im
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@31
PS1, Line 31: from tests.common.test_vector import ImpalaTestDimension
> flake8: F401 'tests.common.test_vector.ImpalaTestDimension' imported but un
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@37
PS1, Line 37: class TestUdfConcurrency(CustomClusterTestSuite):
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@56
PS1, Line 56: a
> flake8: E501 line too long (97 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@84
PS1, Line 84: d
> flake8: E306 expected 1 blank line before a nested definition, found 0
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@121
PS1, Line 121: ;
> flake8: E703 statement ends with a semicolon
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@174
PS1, Line 174: d
> flake8: E306 expected 1 blank line before a nested definition, found 0
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@194
PS1, Line 194: d
> flake8: F841 local variable 'drop_threads' is assigned to but never used
Done
http://gerrit.cloudera.org:8080/#/c/11701/1/tests/custom_cluster/test_udf_concurrency.py@202
PS1, Line 202: ;
> flake8: E703 statement ends with a semicolon
Done
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Oct 2018 17:30:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1077/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Oct 2018 18:00:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7639: Move concurrent UDF tests to a custom cluster test
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11701 )
Change subject: IMPALA-7639: Move concurrent UDF tests to a custom cluster test
......................................................................
Patch Set 1: Code-Review+2
thanks!
--
To view, visit http://gerrit.cloudera.org:8080/11701
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f255823167a4dd807a07276f630ef02435900a3
Gerrit-Change-Number: 11701
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Oct 2018 22:08:24 +0000
Gerrit-HasComments: No