You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2024/03/07 03:26:43 UTC
[PR] [WIP] align gihub workflow run ubuntu version [spark]
panbingkun opened a new pull request, #45416:
URL: https://github.com/apache/spark/pull/45416
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002173352
Hi, @panbingkun and all.
Unfortunately, this seems to break `branch-3.4` and `branch-3.5` CI due to the Python version and package differences. Do you think we can isolate this PR to `master` branch only?
- Python Linter Failure
- https://github.com/apache/spark/actions/runs/8305693061/job/22733444259
```
starting mypy annotations test...
annotations failed mypy checks:
python/pyspark/pandas/plot/matplotlib.py:22: error: Module "matplotlib.axes._base" has no attribute "_process_plot_format" [attr-defined]
Found 1 error in 1 file (checked 425 source files)
1
Error: Process completed with exit code 1.
```
- Pandas Failure
- https://github.com/apache/spark/actions/runs/8305693061/job/22733444673
```
======================================================================
ERROR [15.158s]: test_aggregate (pyspark.pandas.tests.test_groupby_slow.GroupBySlowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/pandas/tests/test_groupby_slow.py", line 287, in test_aggregate
stats_pdf.sort_values(by=[(20, "min"), (20, "max"), (30, "sum")]).reset_index(
File "/usr/local/lib/python3.9/dist-packages/pandas/util/_decorators.py", line 331, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/frame.py", line 6894, in sort_values
keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
File "/usr/local/lib/python3.9/dist-packages/pandas/core/frame.py", line 6894, in <listcomp>
keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 1838, in _get_label_or_level_values
self._check_label_or_level_ambiguity(key, axis=axis)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 1780, in _check_label_or_level_ambiguity
and key in self.axes[axis].names
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
----------------------------------------------------------------------
Ran 15 tests in 174.031s
FAILED (errors=1)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45416:
URL: https://github.com/apache/spark/pull/45416#discussion_r1527732657
##########
.github/workflows/build_and_test.yml:
##########
@@ -52,7 +52,7 @@ on:
jobs:
precondition:
name: Check changes
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
Review Comment:
Yes, correctly. This only reveals the technical depth, @panbingkun .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45416:
URL: https://github.com/apache/spark/pull/45416#discussion_r1527692420
##########
.github/workflows/build_and_test.yml:
##########
@@ -52,7 +52,7 @@ on:
jobs:
precondition:
name: Check changes
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
Review Comment:
From the above analysis, can I conclude that it is not the PR that `caused` the issue, but rather the PR triggered the `regeneration of the image`, which exposed potential problems? Is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002283096
For the record:
1.branch 3.4, Python Linter Failure, https://github.com/apache/spark/actions/runs/8305693061/job/22733444259,
run on: `ubuntu 20.04.6`
<img width="1100" alt="image" src="https://github.com/apache/spark/assets/15246973/cbdf873f-24a9-4636-860b-87c36cba08b8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-1992821619
BTW, we are assuming that `Ubuntu 24.04` next month in this PR, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002769330
What you're doing at https://github.com/apache/spark/pull/45551 would be the way I would test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002432064
Thank you for the analysis, @HyukjinKwon ! I closed the investigation PR, #45545 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-1992942759
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002772657
Okay, I'll continue with this PR https://github.com/apache/spark/pull/45551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002436837
> For [#45416 (comment)](https://github.com/apache/spark/pull/45416#issuecomment-2002352417), it is because we use the cached image from the master branch. So we explicitly overwrite the Python dependencies for linter jobs (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L682-L688). TL;DR, for scheduled jobs, it uses the cached image for `master` branch. In this case, we should pin the dependencies at `master` branch to remove the version diff, cc @panbingkun can you followup this please?
>
> For [#45416 (comment)](https://github.com/apache/spark/pull/45416#issuecomment-2002353686), I commented at [#45545 (comment)](https://github.com/apache/spark/pull/45545#issuecomment-2002355919). TL;DR, In this case, it uses the cached image for the branch itself. It should be fixed if we backport corresponding commits I pointed out in the comment.
Thank you very much for your explanation @HyukjinKwon
Okay, let me try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002286274
From the case given above, the problems that `branch 3.5 run on ubuntu-22.04` & `branch 3.4 run on ubuntu-20.04 `encountered in workflow `Linters, licenses, dependencies and documentation generation` are the same, as shown below:
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002293777
Yes. It looks like that. I have no clue about how this PR can affect the old branch CIs.
So, I'm investigating without reverting this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45416:
URL: https://github.com/apache/spark/pull/45416#discussion_r1527692420
##########
.github/workflows/build_and_test.yml:
##########
@@ -52,7 +52,7 @@ on:
jobs:
precondition:
name: Check changes
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
Review Comment:
From the above analysis, can I conclude that it is not the PR that `caused` the issue, but rather the PR triggered the `regeneration of the image`, which exposed potential problems? Is that correct?
Because there is `no substantial change` in the version of Ubuntu when building the `cache image` on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-1992920305
> BTW, we are assuming that `Ubuntu 24.04` next month in this PR, right?
Yeah, currently `ubuntu-latest` is `ubuntu-22.04`. With the latest stable images (eg.: `ubuntu-24.04`) that GitHub provides, we will use it (`ubuntu-24.04`).
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
<img width="767" alt="image" src="https://github.com/apache/spark/assets/15246973/6649a6f2-ba51-4e7f-8a5c-c129e15d5c91">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002196596
For the record, after testing #45545, I found that we need the following to recover `branch-3.5` which were the previous successful run.
```
matplotlib==3.7.2
torch==2.0.1
torchvision==0.15.2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002353686
The test failures in `branch-3.5` and `branch-3.4` seems because of the behaviour difference of pyarrow version.
```
pyarrow 15.0.1
```
https://github.com/apache/spark/actions/runs/8305690576/job/22746538190 (failed)
vs
```
pyarrow 12.0.1
```
https://github.com/apache/spark/actions/runs/8305690576/job/22746538190 (passed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45416:
URL: https://github.com/apache/spark/pull/45416#discussion_r1527492023
##########
.github/workflows/build_and_test.yml:
##########
@@ -52,7 +52,7 @@ on:
jobs:
precondition:
name: Check changes
- runs-on: ubuntu-22.04
+ runs-on: ubuntu-latest
Review Comment:
Here is my theory.
According to the above @HyukjinKwon analysis, this line could be the reason why this PR regenerates the cached images of branch-3.5 and branch-3.4.
`precondition` job is literally changed here although the actual version is the same, Ubuntu 22.04 == Ubuntu Latest. And, it invokes the following.
https://github.com/apache/spark/blob/714feb7b5aec948c3499ad486dc63ea89a241ffa/.github/workflows/build_and_test.yml#L111-L118
If we look at `Starting job container` stage of `branch-3.5` job, `ghcr.io/apache/apache-spark-ci-image:branch-3.5-8276441035` image has different sha values. It seems to be regenerated.
**BEFORE**: https://github.com/apache/spark/actions/runs/8276441035/job/22645041254
```
Digest: sha256:c97bcef01a653038dba464bc5d947340be4385efa18808b4a3eb813095ea95f9
Status: Downloaded newer image for ghcr.io/apache/apache-spark-ci-image:branch-3.5-8276441035
```
**AFTER**: https://github.com/apache/spark/actions/runs/8276441035/job/22733550856
```
Digest: sha256:a91feba3ada152b513192acb4fa3b2930655e37644f6bad11c30d6c7a6a12383
Status: Downloaded newer image for ghcr.io/apache/apache-spark-ci-image:branch-3.5-8276441035
```
WDTY, @HyukjinKwon and @panbingkun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-1991465881
In all the `YML` files in the workflow, only `two` place `retains` the Ubuntu version of `20.04`, because `databricks/tpcds-kit` depends on it.
https://github.com/apache/spark/blob/6c30525e0d71c707aaaed1be49600b866bced49b/.github/workflows/build_and_test.yml#L857-L858
https://github.com/apache/spark/blob/6c30525e0d71c707aaaed1be49600b866bced49b/.github/workflows/benchmark.yml#L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002357326
For https://github.com/apache/spark/pull/45416#issuecomment-2002352417, it is because we use the cached image from the master branch. So we explicitly overwrite the Python dependencies for linter jobs (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L682-L688). TL;DR, for scheduled jobs, it uses the cached image for `master` branch.
For https://github.com/apache/spark/pull/45416#issuecomment-2002353686, I commented at https://github.com/apache/spark/pull/45545#issuecomment-2002355919. TL;DR, In this case, it uses the cached image for the branch itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002759374
@HyukjinKwon
In addition, I would like to ask for advice. In order to solve the problem of `scheduled jobs`, after pinning the Python dependencies on the `master`, how can I easily test and verify them?
So far, I seem to have not found a suitable method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002769890
Explicitly provide those info specified in https://github.com/apache/spark/blob/master/.github/workflows/build_branch35.yml, and run it within your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002352417
For linter failure at the scheduled job, I think it's because of different Python dependency versions. For example, at least I see one diff
```
Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.9/dist-packages (from mypy==0.920) (4.4.0)
```
https://github.com/apache/spark/actions/runs/8278032972/job/22649628764 (failed)
vs
```
Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.9/dist-packages (from mypy==0.920) (4.10.0)
```
https://github.com/apache/spark/actions/runs/8293729419/job/22699200117 (passed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-2002275806
Let me investigate it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-1992813422
WDYT @Yikun and @dongjoon-hyun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45416:
URL: https://github.com/apache/spark/pull/45416#issuecomment-1990818612
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #45416: [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on
URL: https://github.com/apache/spark/pull/45416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org