You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@predictionio.apache.org by takezoe <gi...@git.apache.org> on 2018/09/22 23:41:07 UTC
[GitHub] predictionio pull request #471: [WIP][PIO-171] Drop Scala 2.10 and Spark 1.6...
GitHub user takezoe opened a pull request:
https://github.com/apache/predictionio/pull/471
[WIP][PIO-171] Drop Scala 2.10 and Spark 1.6 support
- [ ] Build configuration update
- [ ] Documentation update
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/takezoe/predictionio PIO-171_drop-scala2.10
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/predictionio/pull/471.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #471
----
commit 4f7f49d397784bd8be1e1460b3eda88984549890
Author: Naoki Takezoe <ta...@...>
Date: 2018-09-22T23:39:45Z
[PIO-171] Drop Scala 2.10 and Spark 1.6 support
----
---
[GitHub] predictionio pull request #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 supp...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/predictionio/pull/471
---
[GitHub] predictionio issue #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/predictionio/pull/471
@takezoe Looks great. I have no further comments. Thank you!
---
[GitHub] predictionio pull request #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 supp...
Posted by takezoe <gi...@git.apache.org>.
Github user takezoe commented on a diff in the pull request:
https://github.com/apache/predictionio/pull/471#discussion_r220017782
--- Diff: build.sbt ---
@@ -17,24 +17,7 @@
import PIOBuild._
lazy val scalaSparkDepsVersion = Map(
--- End diff --
Yes, but we will add Scala 2.12 support very soon. So keeping this `Map` is better I think.
---
[GitHub] predictionio pull request #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 supp...
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on a diff in the pull request:
https://github.com/apache/predictionio/pull/471#discussion_r219937276
--- Diff: build.sbt ---
@@ -17,24 +17,7 @@
import PIOBuild._
lazy val scalaSparkDepsVersion = Map(
--- End diff --
Maybe we can get rid of this `Map` as well?
---
[GitHub] predictionio pull request #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 supp...
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on a diff in the pull request:
https://github.com/apache/predictionio/pull/471#discussion_r219938608
--- Diff: docs/manual/source/install/index.html.md.erb ---
@@ -24,8 +24,8 @@ limitations under the License.
It is **very important** to meet the minimum version of the following
technologies that power Apache PredictionIO®.
-* Apache Hadoop 2.6.5 (optional, required only if YARN and HDFS are needed)
-* Apache Spark 1.6.3 for Hadoop 2.6
+* Apache Hadoop 2.7.3 (optional, required only if YARN and HDFS are needed)
+* Apache Spark 2.1.1 for Hadoop 2.7
--- End diff --
I wonder if we should put Hadoop 2.6.x/2.7.x, and Spark 2.0+ here. We still run CI on Spark 2.0, and the Hadoop requirement comes mostly from Spark.
---
[GitHub] predictionio issue #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 support
Posted by takezoe <gi...@git.apache.org>.
Github user takezoe commented on the issue:
https://github.com/apache/predictionio/pull/471
I added Spark 2.0.2 / Hadoop 2.6.5 test patterns and updated Spark version in `bin/pio-class.sh` and `tools/src/main/scala/org/apache/predictionio/tools/commands/Management.scala`.
---
[GitHub] predictionio issue #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/predictionio/pull/471
We should also update `bin/pio-class` to require Spark 2.0.0+. It is currently set to 1.6.3.
---
[GitHub] predictionio issue #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/predictionio/pull/471
Merging. Thank you!
---
[GitHub] predictionio issue #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/predictionio/pull/471
Just stumbled upon https://github.com/apache/predictionio/blob/develop/data/src/main/spark-1/org/apache/predictionio/data/SparkVersionDependent.scala. We probably should clean this up as well.
---
[GitHub] predictionio pull request #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 supp...
Posted by takezoe <gi...@git.apache.org>.
Github user takezoe commented on a diff in the pull request:
https://github.com/apache/predictionio/pull/471#discussion_r220019061
--- Diff: docs/manual/source/install/index.html.md.erb ---
@@ -24,8 +24,8 @@ limitations under the License.
It is **very important** to meet the minimum version of the following
technologies that power Apache PredictionIO®.
-* Apache Hadoop 2.6.5 (optional, required only if YARN and HDFS are needed)
-* Apache Spark 1.6.3 for Hadoop 2.6
+* Apache Hadoop 2.7.3 (optional, required only if YARN and HDFS are needed)
+* Apache Spark 2.1.1 for Hadoop 2.7
--- End diff --
I coundn't find Spark 2.0 pattern in CI and default Hadoop version for PredictionIO build is now Hadoop 2.7.3. But in `build.sbt`, Spark 2.0 is still supported and we can specify Hadoop version as 2.6.x. What is minimum supported versions in this case...?
---
[GitHub] predictionio issue #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 support
Posted by takezoe <gi...@git.apache.org>.
Github user takezoe commented on the issue:
https://github.com/apache/predictionio/pull/471
@dszeto Nice catch! I removed the file.
---
[GitHub] predictionio pull request #471: [PIO-171] Drop Scala 2.10 and Spark 1.6 supp...
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on a diff in the pull request:
https://github.com/apache/predictionio/pull/471#discussion_r220067887
--- Diff: docs/manual/source/install/index.html.md.erb ---
@@ -24,8 +24,8 @@ limitations under the License.
It is **very important** to meet the minimum version of the following
technologies that power Apache PredictionIO®.
-* Apache Hadoop 2.6.5 (optional, required only if YARN and HDFS are needed)
-* Apache Spark 1.6.3 for Hadoop 2.6
+* Apache Hadoop 2.7.3 (optional, required only if YARN and HDFS are needed)
+* Apache Spark 2.1.1 for Hadoop 2.7
--- End diff --
How about we try adding an entry to Travis to build against Spark 2.0.2 and Hadoop 2.6.5? If that passes, we could still support that. I do notice that Spark 2.0.x has not received updates for almost 2 years now.
---