You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "vivostar (via GitHub)" <gi...@apache.org> on 2023/02/12 14:36:21 UTC
[GitHub] [bigtop] vivostar opened a new pull request, #1087: BIGTOP-3908 Upgrade Spark Rpm Packages for PySpark Requres Python3
vivostar opened a new pull request, #1087:
URL: https://github.com/apache/bigtop/pull/1087
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
Upgade spark rpm packages for pyspark requires python3, according to the [doc](https://spark.apache.org/docs/3.2.3/api/python/getting_started/quickstart_df.html).
### How was this patch tested?
```bash
./docker-hadoop.sh \
-d \
-dcp\
--create 1 \
--image bigtop/puppet:trunk-rockylinux-8 \
--memory 8g \
-L \
--repo file:///bigtop-home/output \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,spark,hive
```
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
- [x] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] pan3793 commented on a diff in pull request #1087: BIGTOP-3908 Upgrade Spark Rpm Packages for PySpark Requres Python3
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1087:
URL: https://github.com/apache/bigtop/pull/1087#discussion_r1125402632
##########
bigtop-packages/src/rpm/spark/SPECS/spark.spec:
##########
@@ -116,11 +116,7 @@ Server for Spark worker
%package -n %{spark_pkg_name}-python
Summary: Python client for Spark
Group: Development/Libraries
-%if 0%{?rhel} >= 8
-Requires: %{spark_pkg_name}-core = %{version}-%{release}, python2
-%else
-Requires: %{spark_pkg_name}-core = %{version}-%{release}, python
-%endif
+Requires: %{spark_pkg_name}-core = %{version}-%{release}, python36
Review Comment:
Currently, bigtop uses Spark 3.2, but since Spark 3.3, the minimal python version is 3.7, so can we use a higher version to make it more future-proofing?
https://spark.apache.org/docs/3.3.2/api/python/getting_started/install.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on a diff in pull request #1087: BIGTOP-3908 Upgrade Spark Rpm Packages for PySpark Requres Python3
Posted by "vivostar (via GitHub)" <gi...@apache.org>.
vivostar commented on code in PR #1087:
URL: https://github.com/apache/bigtop/pull/1087#discussion_r1148470691
##########
bigtop-packages/src/rpm/spark/SPECS/spark.spec:
##########
@@ -116,11 +116,7 @@ Server for Spark worker
%package -n %{spark_pkg_name}-python
Summary: Python client for Spark
Group: Development/Libraries
-%if 0%{?rhel} >= 8
-Requires: %{spark_pkg_name}-core = %{version}-%{release}, python2
-%else
-Requires: %{spark_pkg_name}-core = %{version}-%{release}, python
-%endif
+Requires: %{spark_pkg_name}-core = %{version}-%{release}, python36
Review Comment:
thanks for your suggestions, I think we can use a higher python3 version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] kevinw66 commented on pull request #1087: BIGTOP-3908 Upgrade Spark Rpm Packages for PySpark Requres Python3
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on PR #1087:
URL: https://github.com/apache/bigtop/pull/1087#issuecomment-1593099048
PySpark stucked in CentOS7.
```
[root@d84cd50ac3f6 /]# pyspark
Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/06/15 13:34:41 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
23/06/15 13:34:42 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
### stucked here
```
BTW, could you also upgrade python for Spark deb packages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] vivostar commented on a diff in pull request #1087: BIGTOP-3908 Upgrade Spark Rpm Packages for PySpark Requres Python3
Posted by "vivostar (via GitHub)" <gi...@apache.org>.
vivostar commented on code in PR #1087:
URL: https://github.com/apache/bigtop/pull/1087#discussion_r1148470691
##########
bigtop-packages/src/rpm/spark/SPECS/spark.spec:
##########
@@ -116,11 +116,7 @@ Server for Spark worker
%package -n %{spark_pkg_name}-python
Summary: Python client for Spark
Group: Development/Libraries
-%if 0%{?rhel} >= 8
-Requires: %{spark_pkg_name}-core = %{version}-%{release}, python2
-%else
-Requires: %{spark_pkg_name}-core = %{version}-%{release}, python
-%endif
+Requires: %{spark_pkg_name}-core = %{version}-%{release}, python36
Review Comment:
yes, we can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org