You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gaogaotiantian (via GitHub)" <gi...@apache.org> on 2026/08/25 07:48:14 UTC
[PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
gaogaotiantian opened a new pull request, #58274:
URL: https://github.com/apache/spark/pull/58274
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
* Replace `pyspark-install` with `pyspark-scheduled`
* Move upstream tests to `pyspark-scheduled`
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
We used to run the pyspark install test separately (scheduled CI and pre-merge) because it was really slow. Then we moved packaging test into the module. Now we have upstream monitoring tests which are not sensitive to pyspark code change but put into `pyspark_core` module because there's no better place for it.
All these tests share a common feature - they are not designed for a specific feature for pyspark. We don't need to run them after every commit because they are not really testing pyspark itself. We could save some valuable time by running them on pre-merge and scheduled CI only. It also makes more sense for the categories.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
CI.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5419425398
> **Proposal:**
>
> 1. Use `pyspark-periodic` to replace `pyspark-install`.
> 2. Make `pyspark-periodic` run on all pre-merge CIs and all scheduled CIs, not just coverage. We are increasing some python version and architecture coverage for `test_install_spark` and `test_import_spark`, not reducing. The cost should not be too large.
>
> WDYT?
My two cents: can we describe these groups based only on what they cover, rather than how/when they run?
* pyspark-upstream: monitors changes in upstream libraries
* pyspark-core: monitors changes related to PySpark installation
This way, we can move them between scheduled/periodic CI, pre-merge CI, and post-merge CI without having to rename the groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #58274:
URL: https://github.com/apache/spark/pull/58274#discussion_r3984716343
##########
.github/workflows/build_and_test.yml:
##########
@@ -155,7 +155,7 @@ jobs:
docs=true
java25=true
else
- pyspark_install=false
+ pyspark_periodic=false
Review Comment:
shall we factor pyspark_periodic out of pyspark? since it doesn't depend on the spark source code and the precompiled artifact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5419344795
Thanks for the review @dongjoon-hyun and @Yicong-Huang.
First of all, it was my oversight that the coverage for pyarrow upstream was reduced - that was not intentional. I fully agree that it should be executed on all platforms + python versions.
The implementation is worth discussing.
I agree `pyspark-scheduled` is not the best name, but we have 4 kinds of tests that we need to include in this mechanism:
1. `test_install_spark` - installation
2. `test_import_spark` - make sure `import pyspark` does not include unnecessary 3rd party libraries
3. All upstream tests to detect if `pyarrow` upgrade changes the current result
4. packaging test which is *not* a python test. It lives in a `sh` file and is currently triggered by detecting whether the module includes `pyspark-install`. We can't easily put it into a pyspark test (maybe we can but it requires serious refactoring). So this does *not* belong to any module.
Is the name `pyspark-periodic` better? Still summarizes the common features of these tests and does not indicate these only runs on scheduled CIs.
One thing I would propose in this PR is to make the current `pyspark-install` tests run on every scheduled Python CI too. Packaging test actually can be impacted by the python version and architecture. We used to have a very slow install test because we are pointing to an archived version. The current install test is pretty fast so we can afford to run it in scheduled CI.
The good thing about running `all these tests on every scheduled CI, is that we don't have to have 2 separate mechanisms for triggering them. We only need a single mechanism to figure out if this is a scheduled CI, or a main commit, which should be pretty simple - just need to check the repo and the event.
**Proposal:**
1. Use `pyspark-periodic` to replace `pyspark-install`.
2. Make `pyspark-periodic` run on all pre-merge CIs and all scheduled CIs, not just coverage. We are increasing some python version and architecture coverage for `test_install_spark` and `test_import_spark`, not reducing. The cost should not be too large.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5416489812
Thank you @gaogaotiantian @Yicong-Huang @zhengruifeng!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5417968562
Thanks for the cleanup! The rename itself looks mechanically complete -- I grepped for any leftover `pyspark-install` / `pyspark_install` reference and found none outside the four files you touched, and the updated `dev/sparktestsupport/utils.py` doctests pass locally.
One thing I'd like to discuss before this goes in.
### Moving the upstream PyArrow tests out of `pyspark-core` drops them from the scheduled version matrix
`pyspark-core` runs in every workflow that passes `"pyspark": "true"`, but `pyspark-scheduled` does not. Since this PR only updates `build_coverage.yml`, the 13 upstream PyArrow test goals stop running in these scheduled workflows:
`build_python_minimum`, `build_python_3.11`, `build_python_3.13`, `build_python_3.14`, `build_python_3.14_nogil`, `build_python_3.12_pandas_3`, `build_python_3.12_classic_only`, `build_java21`, `build_java25`, `build_non_ansi`, `build_uds`, `build_rockdb_as_ui_backend`, `build_codegen_jdk`
They also disappear from the ARM and macOS daily builds, because `.github/workflows/python_hosted_runner_test.yml` names `pyspark-core` directly in its matrix:
```yaml
modules:
- >-
pyspark-sql, pyspark-resource, pyspark-testing
- >-
pyspark-core, pyspark-errors, pyspark-streaming
```
That is the part I'm unsure about. These tests are written specifically for that version matrix -- they carry explicit `LooseVersion(pa.__version__)` / `LooseVersion(pd.__version__)` branches and per-version overrides, and `goldenutils.py` states the goal outright: "so that the same golden file works for multiple versions". `build_python_minimum` (oldest supported PyArrow/pandas) and `build_python_3.14` / `build_python_3.12_pandas_3` (newest) are exactly where an upstream behavior change would surface first, and those are the runs losing the coverage.
So after this PR, "we run these on scheduled CI" effectively means `build_coverage` only.
Two ways forward:
1. Add `"pyspark-scheduled": "true"` to the `jobs` input of the workflows above, and add the module to `python_hosted_runner_test.yml`. The downside is that this also flips `SKIP_PACKAGING=false` (build_and_test.yml, "Reusing precompiled artifact" step), so every one of those runs picks up the slow install/packaging tests too.
2. Split the module instead: keep the install/packaging tests in `pyspark-scheduled` and put the upstream monitoring tests in their own module (e.g. `pyspark-upstream`). Then the version-matrix workflows can opt into the upstream tests without paying for packaging. This also seems closer to the categorization argument in the PR description -- the two groups are "not sensitive to pyspark changes" for quite different reasons, and they want different CI placement.
I lean toward (2), but either is fine with me. If the reduced coverage is intentional, could you note the reasoning in the PR description?
### Minor
The new comment in `build_and_test.yml` is slightly off:
```yaml
# pyspark-scheduled will only run when explicitly requested, it contains tests that are not sensitive to pyspark code changes.
```
It is not only when explicitly requested -- on fork pre-merge CI it is still triggered by `is-changed.py` whenever `python/` changes, which is why the `Build modules: pyspark-scheduled` job ran on this very PR. The previous wording ("when it's changed or explicitly requested") was accurate. While there, the line is 135 characters; wrapping it would match the surrounding comments.
Same nit on the name: `scheduled` reads as "runs on the scheduled CI", but in practice this module runs mostly on pre-merge. Not a blocker, though if you go with the module split above, something like `pyspark-packaging` / `pyspark-upstream` would describe each group more directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5418046289
I actually would agree with @dongjoon-hyun that if we can have more groups like `pyspark-packaging` / `pyspark-upstream` and then decide which group to be put into pre-merge CI, which group to be put into scheduled CI, then it would be easier to move tests around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #58274:
URL: https://github.com/apache/spark/pull/58274#discussion_r3985609840
##########
.github/workflows/build_and_test.yml:
##########
@@ -155,7 +155,7 @@ jobs:
docs=true
java25=true
else
- pyspark_install=false
+ pyspark_periodic=false
Review Comment:
By “factor out,” I meant moving it out of the precompiled-artifact-dependent PySpark CI matrix, not moving the test files. These tests could run in a lightweight Python-only job without waiting for Spark precompile, downloading and extracting its large artifact, or setting up Java/SBT.
However, I missed that `pyspark-periodic` also enables `dev/run-pip-tests`, which packages assembly jars and runs `spark-submit`. That part does need the compiled artifact. A clean split would move the upstream/import/install tests to the lightweight job while keeping the packaging test artifact-dependent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5437492939
> I'm not against the idea to have a new mechanism to label where these tests should run, but that's a much bigger change.
ok we could defer this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #58274:
URL: https://github.com/apache/spark/pull/58274#discussion_r4009211386
##########
.github/workflows/build_and_test.yml:
##########
@@ -155,7 +155,7 @@ jobs:
docs=true
java25=true
else
- pyspark_install=false
+ pyspark_periodic=false
Review Comment:
I think `import pyspark` requires the pre-compiled artifact right? Maybe we can write the test in a way that it does not even rely on `pyspark`. However, for pre-merge tests, we have to compile the thing anyway. For scheduled CIs, it's just some extra work for each heavy CI. I don't think we gain a lot if we have a separate CI for it. Structure wise maybe it's better? But we still want to test against the Python matrix and different OSes, so this is probably the easier way to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-scheduled [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5419683392
> This way, we can move them between scheduled/periodic CI, pre-merge CI, and post-merge CI without having to rename the groups.
We don't really have a way to move tests between these CIs. Whether the CI runs relies on the module name (at least for now). The CI knows it should only run `pyspark-install` on coverage, so it only runs on coverage and pre-merge.
I'm not against the idea to have a new mechanism to label where these tests should run, but that's a much bigger change.
For our current infra, asking the CI to run `pyspark-periodic` for scheduled CI makes sense. The alternative would be asking the CI to run `pyspark-upstream` and `pyspark-install` on scheduled CI.
Feel free to take a look at the `pyspark` section of `build_and_test` and propose your idea. Do you want something like
```yaml
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark-periodic != 'true' && 'pyspark-upstream, pyspark-install' }}
```
that could be another option - having a switch and a list of modules under it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5845124695
> My two cents: can we describe these groups based only on what they cover, rather than how/when they run?
Just noticed this change and came here to agree with this idea. Modules are supposed to describe test topics and dependencies, not schedules.
It sounds like we already agree with this basic principle and just need a solution that enables us to express execution preferences on top of or alongside the module definitions. I'll look into this unless @gaogaotiantian gets to it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5802793911
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/63941fc2cb971023b94c4bc04f32fdf8830385fb
- merged into branch-4.x https://github.com/apache/spark/commit/4069821fafdf513701f0f437aaa7b2cf0666f36a
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian closed pull request #58274: [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic
URL: https://github.com/apache/spark/pull/58274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #58274:
URL: https://github.com/apache/spark/pull/58274#discussion_r3984923657
##########
.github/workflows/build_and_test.yml:
##########
@@ -155,7 +155,7 @@ jobs:
docs=true
java25=true
else
- pyspark_install=false
+ pyspark_periodic=false
Review Comment:
Could you elaborate on that? The tests still live in pyspark. How do we factor this out? I did not fully get the idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58989][PYTHON][TESTS] Replace pyspark-install with pyspark-periodic [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #58274:
URL: https://github.com/apache/spark/pull/58274#issuecomment-5625379053
I updated this change so
* The module is renamed to `pyspark-periodic`
* The module will run on pre-merge CIs and scheduled CIs.
I agree we can have better ways to label each module where it should run (more and more like what bazel is supposed to do). I can work on it when I have some time. For now we can benefit from not having to run upstream tests on every post merge CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org