You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2024/03/17 14:15:54 UTC
[PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
panbingkun opened a new pull request, #45551:
URL: https://github.com/apache/spark/pull/45551
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2002624940
I backported #42897 to branch-3.5 and branch-3.4. Could you rebase this PR to the `master` branch once more, @panbingkun .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2002613124
Thank you, @panbingkun . Please let me know when this PR is ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1529878116
##########
.github/workflows/build_and_test.yml:
##########
@@ -438,11 +268,21 @@ jobs:
curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
rm miniconda.sh
+ - name: Install Python test dependencies for branch-3.4
Review Comment:
In order to `pass` the pyspark testing of `branch-3.4` and `branch-3.5`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1527751510
##########
.github/workflows/build_and_test.yml:
##########
@@ -801,53 +803,53 @@ jobs:
- java: 21
os: ubuntu-latest
- java: 21
- os: macos-14
+ os: macos-14
runs-on: ${{ matrix.os }}
timeout-minutes: 300
steps:
- - name: Checkout Spark repository
- uses: actions/checkout@v4
- with:
- fetch-depth: 0
- repository: apache/spark
- ref: ${{ inputs.branch }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
- - name: Cache Scala, SBT and Maven
- uses: actions/cache@v4
- with:
- path: |
- build/apache-maven-*
- build/scala-*
- build/*.jar
- ~/.sbt
- key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-
- - name: Cache Maven local repository
- uses: actions/cache@v4
- with:
- path: ~/.m2/repository
- key: java${{ matrix.java }}-maven-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- java${{ matrix.java }}-maven-
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v4
- with:
- distribution: zulu
- java-version: ${{ matrix.java }}
- - name: Build with Maven
- run: |
- export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- export JAVA_VERSION=${{ matrix.java }}
- # It uses Maven's 'install' intentionally, see https://github.com/apache/spark/pull/26414.
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} install
- rm -rf ~/.m2/repository/org/apache/spark
+ - name: Checkout Spark repository
Review Comment:
NVM. This is an outdated comment about the previous commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2003630578
Finally, I can validate the logic of the `branch` in the pr submitted by `master`. Currently, I am `pinning` some python dependency library versions and `verifying` them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2005123258
```
Warning, treated as error:
/__w/spark/spark/python/docs/source/reference/api/pyspark.ml.Estimator.rst:60:autosummary: failed to import Estimator.uid.
Possible hints:
* AttributeError: type object 'Estimator' has no attribute 'uid'
* ModuleNotFoundError: No module named 'Estimator'
* ImportError:
* AttributeError: type object 'Estimator' has no attribute 'Estimator'
* KeyError: 'Estimator'
* ModuleNotFoundError: No module named 'pyspark.ml.Estimator'
make: *** [Makefile:35: html] Error 2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1529875660
##########
.github/workflows/build_and_test.yml:
##########
@@ -365,7 +202,7 @@ jobs:
pyspark-pandas-connect-part3
env:
MODULES_TO_TEST: ${{ matrix.modules }}
- PYTHON_TO_TEST: 'python3.9'
+ PYTHON_TO_TEST: ''
Review Comment:
Same as the value specified in `branch-3.5.yml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1529878679
##########
.github/workflows/build_and_test.yml:
##########
@@ -438,11 +268,21 @@ jobs:
curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
rm miniconda.sh
+ - name: Install Python test dependencies for branch-3.4
+ if: matrix.branch == 'branch-3.4'
+ run: |
+ python3.9 -m pip install 'numpy==1.24.4' 'pandas<=2.0.3''pyarrow==12.0.1' 'matplotlib==3.7.2' 'torch==2.0.1' 'torchvision==0.15.2' 'scikit-learn==1.1.*'
+ - name: Install Python test dependencies for branch-3.5
+ if: matrix.branch == 'branch-3.5'
+ run: |
+ python3.9 -m pip install 'numpy==1.25.1' 'pandas<=2.0.3' 'pyarrow==12.0.1' 'matplotlib==3.7.2' 'torch==2.0.1' 'torchvision==0.15.2' 'scikit-learn==1.1.*'
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
+ export SCALA_PROFILE="scala2.13"
+ unset GITHUB_ACTIONS
Review Comment:
hack git compare
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2005531332
So far, the tests related to the `PySpark` in `branch-3.5` have `passed`, and the issue of `docs build` is currently being resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2002870795
I temporarily removed some unrelated tests to make it faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2002668610
> I backported #42897 to branch-3.5 and branch-3.4. Could you rebase this PR to the `master` branch once more, @panbingkun .
Okay, I will verify it first today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1527750689
##########
.github/workflows/build_and_test.yml:
##########
@@ -801,53 +803,53 @@ jobs:
- java: 21
os: ubuntu-latest
- java: 21
- os: macos-14
+ os: macos-14
runs-on: ${{ matrix.os }}
timeout-minutes: 300
steps:
- - name: Checkout Spark repository
- uses: actions/checkout@v4
- with:
- fetch-depth: 0
- repository: apache/spark
- ref: ${{ inputs.branch }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
- - name: Cache Scala, SBT and Maven
- uses: actions/cache@v4
- with:
- path: |
- build/apache-maven-*
- build/scala-*
- build/*.jar
- ~/.sbt
- key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-
- - name: Cache Maven local repository
- uses: actions/cache@v4
- with:
- path: ~/.m2/repository
- key: java${{ matrix.java }}-maven-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- java${{ matrix.java }}-maven-
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v4
- with:
- distribution: zulu
- java-version: ${{ matrix.java }}
- - name: Build with Maven
- run: |
- export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- export JAVA_VERSION=${{ matrix.java }}
- # It uses Maven's 'install' intentionally, see https://github.com/apache/spark/pull/26414.
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} install
- rm -rf ~/.m2/repository/org/apache/spark
+ - name: Checkout Spark repository
Review Comment:
This new indentation looks like a mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1529876143
##########
.github/workflows/build_and_test.yml:
##########
@@ -382,17 +219,10 @@ jobs:
with:
fetch-depth: 0
repository: apache/spark
- ref: ${{ inputs.branch }}
+ ref: ${{ matrix.branch }}
Review Comment:
Only for test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2006193011
This PR is basically successful, and I will slightly organize it and submit it as a separate PR. This PR allows me to keep it as a reference for future testing of the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1529888833
##########
.github/workflows/build_and_test.yml:
##########
@@ -684,32 +438,18 @@ jobs:
run: |
# SPARK-44554: Copy from https://github.com/apache/spark/blob/a05c27e85829fe742c1828507a1fd180cdc84b54/.github/workflows/build_and_test.yml#L571-L578
# Should delete this section after SPARK 3.4 EOL.
- python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.920' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc 'jinja2<3.0.0' 'black==22.6.0'
+ python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.920' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'numpy==1.25.1' 'pyarrow==12.0.1' numpydoc 'jinja2<3.0.0' 'black==22.6.0' 'pandas<=2.0.3' 'matplotlib==3.7.2' 'torch==2.0.1' 'torchvision==0.15.2'
python3.9 -m pip install 'pandas-stubs==1.2.0.53' ipython 'grpcio==1.48.1' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0'
- name: Install Python linter dependencies for branch-3.5
- if: inputs.branch == 'branch-3.5'
run: |
# SPARK-45212: Copy from https://github.com/apache/spark/blob/555c8def51e5951c7bf5165a332795e9e330ec9d/.github/workflows/build_and_test.yml#L631-L638
# Should delete this section after SPARK 3.5 EOL.
- python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.982' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc 'jinja2<3.0.0' 'black==22.6.0'
+ python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.982' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'numpy==1.25.1' 'pyarrow==12.0.1' numpydoc 'jinja2<3.0.0' 'black==22.6.0' 'pandas<=2.0.3' 'matplotlib==3.7.2' 'torch==2.0.1' 'torchvision==0.15.2'
Review Comment:
For fixing as
```
/usr/local/lib/python3.9/dist-packages/torch/_dynamo/mutation_guard.py:1: error: disable_error_code: Invalid error code(s): method-assign [misc]
/usr/local/lib/python3.9/dist-packages/torch/_dynamo/eval_frame.py:1: error: disable_error_code: Invalid error code(s): method-assign [misc]
/usr/local/lib/python3.9/dist-packages/torch/_dynamo/debug_utils.py:1: error: disable_error_code: Invalid error code(s): method-assign [misc]
python/pyspark/pandas/plot/matplotlib.py:23: error: Module "matplotlib.axes._base" has no attribute "_process_plot_format" [attr-defined]
Found 4 errors in 4 files (checked 688 source files)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun closed pull request #45551: [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5
URL: https://github.com/apache/spark/pull/45551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1527754324
##########
.github/workflows/build_and_test.yml:
##########
@@ -801,53 +803,53 @@ jobs:
- java: 21
os: ubuntu-latest
- java: 21
- os: macos-14
+ os: macos-14
runs-on: ${{ matrix.os }}
timeout-minutes: 300
steps:
- - name: Checkout Spark repository
- uses: actions/checkout@v4
- with:
- fetch-depth: 0
- repository: apache/spark
- ref: ${{ inputs.branch }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
- - name: Cache Scala, SBT and Maven
- uses: actions/cache@v4
- with:
- path: |
- build/apache-maven-*
- build/scala-*
- build/*.jar
- ~/.sbt
- key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-
- - name: Cache Maven local repository
- uses: actions/cache@v4
- with:
- path: ~/.m2/repository
- key: java${{ matrix.java }}-maven-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- java${{ matrix.java }}-maven-
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v4
- with:
- distribution: zulu
- java-version: ${{ matrix.java }}
- - name: Build with Maven
- run: |
- export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- export MAVEN_CLI_OPTS="--no-transfer-progress"
- export JAVA_VERSION=${{ matrix.java }}
- # It uses Maven's 'install' intentionally, see https://github.com/apache/spark/pull/26414.
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} install
- rm -rf ~/.m2/repository/org/apache/spark
+ - name: Checkout Spark repository
Review Comment:
Yeah, I'm trying to find a way to verify it faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45551:
URL: https://github.com/apache/spark/pull/45551#issuecomment-2005543983
Before the pr https://github.com/apache/spark/pull/44012 was `backport` to `branch-3.5` or `branch-3.4`, there was a issue when sphinx versions >= `3.1.0`.
But in the file `build_and_test.yml` of our `master`, we set `sphinx==4.5.0`
https://github.com/apache/spark/blob/acf17fd67217941891fde0851b351b99e55d7c4a/.github/workflows/build_and_test.yml#L756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45551:
URL: https://github.com/apache/spark/pull/45551#discussion_r1529874542
##########
.github/workflows/build_and_test.yml:
##########
@@ -341,7 +176,9 @@ jobs:
fail-fast: false
matrix:
java:
- - ${{ inputs.java }}
+ - 8
Review Comment:
only for test branch-3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org