You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2026/03/04 04:25:58 UTC

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24066


Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................

IMPALA-14776 (part 3): Fix misc resource leaks

This fixes a variety of misc resource leaks and other warnings.
This is purely about reducing the noise, so these are not
fixing issues that would impact build stability.

There are a few different categories:
1. Leaked open files
 - There are a couple locations that open /dev/null manually
   to use with subprocess. This switches them to subprocess.DEVNULL
 - Otherwise, use a context manager
2. Leaked thread pools - Manually call terminate at the end
3. Leaked processes - Add a wait() call after killing them
4. Warnings about deprecated pyparsing APIs - switched to the
   new API in db_connection.py
5. Warnings about invalid escapes - used raw strings for regex
6. Warnings from pytest failing to collect tests from
   TestLastDdlTimeUpdate's TestHelper
 - Renamed TestHelper to Helper so pytest doesn't try to find
   tests in it
7. A few minor socket leaks

Testing:
 - Ran a core job

Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
---
M tests/common/impala_test_suite.py
M tests/comparison/db_connection.py
M tests/custom_cluster/test_concurrent_ddls.py
M tests/custom_cluster/test_local_catalog.py
M tests/custom_cluster/test_shell_interactive.py
M tests/custom_cluster/test_topic_update_frequency.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_last_ddl_time_update.py
M tests/performance/query_executor.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/stress/stress_util.py
M tests/util/auto_scaler.py
M tests/util/compute_table_stats.py
15 files changed, 30 insertions(+), 13 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/24066/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13193/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 20:53:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/21856/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 04:52:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1:

Test failure is not related


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 17:42:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1: Code-Review+1

(2 comments)

Useful cleanup. A bunch of the pool termination should be in a finally block - or use a with-clause - to ensure cleanup on test failure. Still helpful to clean up the warnings for successful runs.

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py
File tests/custom_cluster/test_concurrent_ddls.py:

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py@198
PS1, Line 198:     pool.terminate()
Should this be in a finally block? "assert False" could fail the test and leave the pool dangling.


http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_local_catalog.py@499
PS1, Line 499:     t.terminate()
Should be in a finally block in case imap_unordered throws an error.



-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 22:16:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1:

(1 comment)

I'm going to go ahead with this.

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py
File tests/custom_cluster/test_concurrent_ddls.py:

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py@198
PS1, Line 198:     pool.terminate()
> IMO we shouldn't add noise with the "with" unless leaking it cause issues.
Fixtures are good for standardized things, but my impression is that it is harder when you have a parameter that you want to pass in (e.g. the number of threads in the thread pool).



-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 20:11:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13187/


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 08:42:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 01:07:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................

IMPALA-14776 (part 3): Fix misc resource leaks

This fixes a variety of misc resource leaks and other warnings.
This is purely about reducing the noise, so these are not
fixing issues that would impact build stability.

There are a few different categories:
1. Leaked open files
 - There are a couple locations that open /dev/null manually
   to use with subprocess. This switches them to subprocess.DEVNULL
 - Otherwise, use a context manager
2. Leaked thread pools - Manually call terminate at the end
3. Leaked processes - Add a wait() call after killing them
4. Warnings about deprecated pyparsing APIs - switched to the
   new API in db_connection.py
5. Warnings about invalid escapes - used raw strings for regex
6. Warnings from pytest failing to collect tests from
   TestLastDdlTimeUpdate's TestHelper
 - Renamed TestHelper to Helper so pytest doesn't try to find
   tests in it
7. A few minor socket leaks

Testing:
 - Ran a core job

Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Reviewed-on: http://gerrit.cloudera.org:8080/24066
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
---
M tests/common/impala_test_suite.py
M tests/comparison/db_connection.py
M tests/custom_cluster/test_concurrent_ddls.py
M tests/custom_cluster/test_local_catalog.py
M tests/custom_cluster/test_shell_interactive.py
M tests/custom_cluster/test_topic_update_frequency.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_last_ddl_time_update.py
M tests/performance/query_executor.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/stress/stress_util.py
M tests/util/auto_scaler.py
M tests/util/compute_table_stats.py
15 files changed, 30 insertions(+), 13 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Csaba Ringhofer: Looks good to me, but someone else must approve
  Michael Smith: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13187/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Mar 2026 04:26:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1: Code-Review+2

I'm fine with this as-is.


-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 16:56:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14776 (part 3): Fix misc resource leaks

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24066 )

Change subject: IMPALA-14776 (part 3): Fix misc resource leaks
......................................................................


Patch Set 1: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py
File tests/custom_cluster/test_concurrent_ddls.py:

http://gerrit.cloudera.org:8080/#/c/24066/1/tests/custom_cluster/test_concurrent_ddls.py@198
PS1, Line 198:     pool.terminate()
> Should this be in a finally block? "assert False" could fail the test and l
IMO we shouldn't add noise with the "with" unless leaking it cause issues.
The most sympathetic solution to me would be to add a threadpool fixture to have both minimal noise and proper cleanup



-- 
To view, visit http://gerrit.cloudera.org:8080/24066
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Gerrit-Change-Number: 24066
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 08:13:59 +0000
Gerrit-HasComments: Yes