You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2024/10/16 02:06:19 UTC

[PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

zhengruifeng opened a new pull request, #48489:
URL: https://github.com/apache/spark/pull/48489

   ### What changes were proposed in this pull request?
   Restore `scipy` installation in dockerfile
   
   
   ### Why are the changes needed?
   `scipy` actually depends on `gfortran`, but `apt-get remove --purge -y 'gfortran-11'` broke this dependency.
   
   
   ### Does this PR introduce _any_ user-facing change?
   no
   
   ### How was this patch tested?
   manually check with the first commit:
   move `apt-get remove --purge -y 'gfortran-11'` ahead of `scipy` installation, then the installation fails with
   ```
   #18 394.3 Collecting scipy
   #18 394.4   Downloading scipy-1.13.1.tar.gz (57.2 MB)
   #18 395.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.2/57.2 MB 76.7 MB/s eta 0:00:00
   #18 401.3   Installing build dependencies: started
   #18 410.5   Installing build dependencies: finished with status 'done'
   #18 410.5   Getting requirements to build wheel: started
   #18 410.7   Getting requirements to build wheel: finished with status 'done'
   #18 410.7   Installing backend dependencies: started
   #18 411.8   Installing backend dependencies: finished with status 'done'
   #18 411.8   Preparing metadata (pyproject.toml): started
   #18 414.9   Preparing metadata (pyproject.toml): finished with status 'error'
   #18 414.9   error: subprocess-exited-with-error
   #18 414.9   
   #18 414.9   × Preparing metadata (pyproject.toml) did not run successfully.
   #18 414.9   │ exit code: 1
   #18 414.9   ╰─> [42 lines of output]
   #18 414.9       + meson setup /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d/.mesonpy-xqfvs4ek -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d/.mesonpy-xqfvs4ek/meson-python-native-file.ini
   #18 414.9       The Meson build system
   #18 414.9       Version: 1.5.2
   #18 414.9       Source dir: /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d
   #18 414.9       Build dir: /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d/.mesonpy-xqfvs4ek
   #18 414.9       Build type: native build
   #18 414.9       Project name: scipy
   #18 414.9       Project version: 1.13.1
   #18 414.9       C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
   #18 414.9       C linker for the host machine: cc ld.bfd 2.38
   #18 414.9       C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
   #18 414.9       C++ linker for the host machine: c++ ld.bfd 2.38
   #18 414.9       Cython compiler for the host machine: cython (cython 3.0.11)
   #18 414.9       Host machine cpu family: x86_64
   #18 414.9       Host machine cpu: x86_64
   #18 414.9       Program python found: YES (/usr/local/bin/pypy3)
   #18 414.9       Run-time dependency python found: YES 3.9
   #18 414.9       Program cython found: YES (/tmp/pip-build-env-v_vnvt3h/overlay/bin/cython)
   #18 414.9       Compiler for C supports arguments -Wno-unused-but-set-variable: YES
   #18 414.9       Compiler for C supports arguments -Wno-unused-function: YES
   #18 414.9       Compiler for C supports arguments -Wno-conversion: YES
   #18 414.9       Compiler for C supports arguments -Wno-misleading-indentation: YES
   #18 414.9       Library m found: YES
   #18 414.9       
   #18 414.9       ../meson.build:78:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']]
   #18 414.9       The following exception(s) were encountered:
   #18 414.9       Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'"
   #18 414.9       Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'"
   #18 414.9       Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'"
   #18 414.9       Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'"
   #18 414.9       Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'"
   #18 414.9       Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'"
   #18 414.9       Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'"
   #18 414.9       Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'"
   #18 414.9       Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'"
   #18 414.9       Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'"
   #18 414.9       Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'"
   #18 414.9       Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'"
   #18 414.9       Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'"
   #18 414.9       Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'"
   #18 414.9       
   #18 414.9       A full log can be found at /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984[4155](https://github.com/zhengruifeng/spark/actions/runs/11357130578/job/31589506939#step:7:4161)33ae9079d/.mesonpy-xqfvs4ek/meson-logs/meson-log.txt
   #18 414.9       [end of output]
   ```
   
   see https://github.com/zhengruifeng/spark/actions/runs/11357130578/job/31589506939
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   no


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #48489:
URL: https://github.com/apache/spark/pull/48489#issuecomment-2416143556

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #48489:
URL: https://github.com/apache/spark/pull/48489#discussion_r1802236313


##########
dev/infra/Dockerfile:
##########
@@ -152,6 +152,6 @@ RUN python3.13 -m pip install lxml numpy>=2.1 && \
     python3.13 -m pip cache purge
 
 # Remove unused installation packages to free up disk space
-RUN apt-get remove --purge -y 'gfortran-11' 'humanity-icon-theme' 'nodejs-doc' || true

Review Comment:
   remove `|| true` to avoid silent failure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #48489:
URL: https://github.com/apache/spark/pull/48489#issuecomment-2415915153

   Just wanted to add that dev/create-release/spark-rm/Dockerfile also has gfortran installed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #48489: [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile
URL: https://github.com/apache/spark/pull/48489


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48489:
URL: https://github.com/apache/spark/pull/48489#issuecomment-2418313146

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #48489:
URL: https://github.com/apache/spark/pull/48489#issuecomment-2415578067

   cc @dongjoon-hyun and @panbingkun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org