You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/04/20 10:02:00 UTC
[jira] [Work logged] (BEAM-14334) Avoid using forkEvery in Spark runner tests
[ https://issues.apache.org/jira/browse/BEAM-14334?focusedWorklogId=759084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-759084 ]
ASF GitHub Bot logged work on BEAM-14334:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Apr/22 10:01
Start Date: 20/Apr/22 10:01
Worklog Time Spent: 10m
Work Description: mosche opened a new pull request, #17406:
URL: https://github.com/apache/beam/pull/17406
Usage of `forkEvery 1` is typically a strong sign of poor quality / bad code and should be avoided:
- It often hides resource leaks, either in code or worse, potentially in the runner itself.
- It significantly impacts performance when running tests.
This PR introduces new JUnit rules to better manage the lifecyle of `SparkContext` or `SparkContextOptions` and allows to safely reuse those for all tests of a suite where applicable. The previous `ReuseSparkContextRule` leaks the SparkContext and was removed. Besides, with `forkEvery 1` trying to reuse the `SparkContext` is absolutely pointless ...
There's might be more issues hidden I didn't discover locally ... I'll keep a look.
This also reduce the runtime of tests to ~30 secs from 1:30+ mins.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
Issue Time Tracking
-------------------
Worklog Id: (was: 759084)
Remaining Estimate: 0h
Time Spent: 10m
> Avoid using forkEvery in Spark runner tests
> -------------------------------------------
>
> Key: BEAM-14334
> URL: https://issues.apache.org/jira/browse/BEAM-14334
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark, testing
> Reporter: Moritz Mack
> Assignee: Moritz Mack
> Priority: P2
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Usage of *{color:#FF0000}forkEvery 1{color}* is typically a strong sign of poor quality / bad code and should be avoided:
> * It significantly impacts performance when running tests.
> * And it often hides resource leaks, either in code or worse in the runner itself.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)