You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2026/03/04 04:25:58 UTC
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24066
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
IMPALA-14776 (part 3): Fix misc resource leaks
This fixes a variety of misc resource leaks and other warnings.
This is purely about reducing the noise, so these are not
fixing issues that would impact build stability.
There are a few different categories:
1. Leaked open files
- There are a couple locations that open /dev/null manually
to use with subprocess. This switches them to subprocess.DEVNULL
- Otherwise, use a context manager
2. Leaked thread pools - Manually call terminate at the end
3. Leaked processes - Add a wait() call after killing them
4. Warnings about deprecated pyparsing APIs - switched to the
new API in db_connection.py
5. Warnings about invalid escapes - used raw strings for regex
6. Warnings from pytest failing to collect tests from
TestLastDdlTimeUpdate's TestHelper
- Renamed TestHelper to Helper so pytest doesn't try to find
tests in it
7. A few minor socket leaks
Testing:
- Ran a core job
Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
---
M tests/common/impala_test_suite.py
M tests/comparison/db_connection.py
M tests/custom_cluster/test_concurrent_ddls.py
M tests/custom_cluster/test_local_catalog.py
M tests/custom_cluster/test_shell_interactive.py
M tests/custom_cluster/test_topic_update_frequency.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_last_ddl_time_update.py
M tests/performance/query_executor.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/stress/stress_util.py
M tests/util/auto_scaler.py
M tests/util/compute_table_stats.py
15 files changed, 30 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/24066/1
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13193/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 20:53:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/21856/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 04:52:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1:
Test failure is not related
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 17:42:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
Useful cleanup. A bunch of the pool termination should be in a finally block - or use a with-clause - to ensure cleanup on test failure. Still helpful to clean up the warnings for successful runs.
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py
File tests/custom_cluster/test_concurrent_ddls.py:
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py@198
PS1, Line 198: pool.terminate()
Should this be in a finally block? "assert False" could fail the test and leave the pool dangling.
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_local_catalog.py@499
PS1, Line 499: t.terminate()
Should be in a finally block in case imap_unordered throws an error.
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 22:16:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1:
(1 comment)
I'm going to go ahead with this.
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py
File tests/custom_cluster/test_concurrent_ddls.py:
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py@198
PS1, Line 198: pool.terminate()
> IMO we shouldn't add noise with the "with" unless leaking it cause issues.
Fixtures are good for standardized things, but my impression is that it is harder when you have a parameter that you want to pass in (e.g. the number of threads in the thread pool).
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 20:11:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13187/
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 08:42:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 01:07:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
IMPALA-14776 (part 3): Fix misc resource leaks
This fixes a variety of misc resource leaks and other warnings.
This is purely about reducing the noise, so these are not
fixing issues that would impact build stability.
There are a few different categories:
1. Leaked open files
- There are a couple locations that open /dev/null manually
to use with subprocess. This switches them to subprocess.DEVNULL
- Otherwise, use a context manager
2. Leaked thread pools - Manually call terminate at the end
3. Leaked processes - Add a wait() call after killing them
4. Warnings about deprecated pyparsing APIs - switched to the
new API in db_connection.py
5. Warnings about invalid escapes - used raw strings for regex
6. Warnings from pytest failing to collect tests from
TestLastDdlTimeUpdate's TestHelper
- Renamed TestHelper to Helper so pytest doesn't try to find
tests in it
7. A few minor socket leaks
Testing:
- Ran a core job
Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Reviewed-on: http://gerrit.cloudera.org:8080/24066
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
---
M tests/common/impala_test_suite.py
M tests/comparison/db_connection.py
M tests/custom_cluster/test_concurrent_ddls.py
M tests/custom_cluster/test_local_catalog.py
M tests/custom_cluster/test_shell_interactive.py
M tests/custom_cluster/test_topic_update_frequency.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_last_ddl_time_update.py
M tests/performance/query_executor.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/stress/stress_util.py
M tests/util/auto_scaler.py
M tests/util/compute_table_stats.py
15 files changed, 30 insertions(+), 13 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Csaba Ringhofer: Looks good to me, but someone else must approve
Michael Smith: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13187/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 04:26:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1: Code-Review+2
I'm fine with this as-is.
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 16:56:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )
Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py
File tests/custom_cluster/test_concurrent_ddls.py:
http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py@198
PS1, Line 198: pool.terminate()
> Should this be in a finally block? "assert False" could fail the test and l
IMO we shouldn't add noise with the "with" unless leaking it cause issues.
The most sympathetic solution to me would be to add a threadpool fixture to have both minimal noise and proper cleanup
--
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 08:13:59 +0000
Gerrit-HasComments: Yes