You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@predictionio.apache.org by marevol <gi...@git.apache.org> on 2017/09/21 03:18:54 UTC
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
GitHub user marevol opened a pull request:
https://github.com/apache/incubator-predictionio/pull/436
[PIO-125] Add Spark 2.2 support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marevol/incubator-predictionio spark22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/436.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #436
----
commit 7ed1e9cb5282def385f49c0d01b8d4561d663bd4
Author: Shinsuke Sugaya <sh...@apache.org>
Date: 2017-09-20T10:47:28Z
add spark 2.2
----
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
LGTM. Merging. Thanks @marevol !
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
Spark 2.2 has guava dependency problem and [SPARK-21549](https://issues.apache.org/jira/browse/SPARK-21549).
I added a workaround for SPARK-21549.
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-predictionio/pull/436
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
For test failures on Travis, I think it's Travis problem. [My Travis](https://travis-ci.org/jpioug/incubator-predictionio/builds/282978334) result is passed.
Spark 2.2 seems to break Guava dependency...
@shimamoto looked into this problem.
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by shimamoto <gi...@git.apache.org>.
Github user shimamoto commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
It is not supposed to need the new Guava dependency because Spark has Guava dependency.
But if this dependency isn't included, compile error occurs at data project.
When I investigated the cause, I discovered Spark 2.1.1 and Spark 2.2.0 were different Guava dependencies.
- Spark 2.1.1 : com.google.guava:guava:14.0.1
- Spark 2.2.0 : com.google.guava:guava:11.0.2
It is expected guava:14.0.1 in Spark 2.2 intrinsically.
The root cause is probably this:
https://github.com/sbt/sbt/issues/2861
The Apache Curator version referenced in Spark has changed in 2.2.0.
This has been thought to have some effect.
https://github.com/apache/spark/blob/v2.1.1/pom.xml#L130
https://github.com/apache/spark/blob/v2.2.0/pom.xml#L126
https://github.com/apache/curator/blob/2.4.0/pom.xml#L307
https://github.com/apache/curator/blob/apache-curator-2.6.0/pom.xml#L424
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
@marevol Apache installed `docker-compose` on their Jenkins finally. We can move these tests there for faster build time.
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
Re-run Travis.
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by dszeto <gi...@git.apache.org>.
Github user dszeto commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
Hey @marevol , any idea with the failing test? I'm wondering if the new Guava dependency is not working fine with older Hadoop/Spark.
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
GitHub user marevol reopened a pull request:
https://github.com/apache/incubator-predictionio/pull/436
[PIO-125] Add Spark 2.2 support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marevol/incubator-predictionio spark22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/436.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #436
----
commit 7ed1e9cb5282def385f49c0d01b8d4561d663bd4
Author: Shinsuke Sugaya <sh...@apache.org>
Date: 2017-09-20T10:47:28Z
add spark 2.2
----
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
Re-run Travis.
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
GitHub user marevol reopened a pull request:
https://github.com/apache/incubator-predictionio/pull/436
[PIO-125] Add Spark 2.2 support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marevol/incubator-predictionio spark22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/436.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #436
----
commit 7ed1e9cb5282def385f49c0d01b8d4561d663bd4
Author: Shinsuke Sugaya <sh...@apache.org>
Date: 2017-09-20T10:47:28Z
add spark 2.2
commit a657cb50ca64998df3fe9ec01468421bd7dff67f
Author: Shinsuke Sugaya <sh...@apache.org>
Date: 2017-10-04T13:49:35Z
add mapreduce.output.fileoutputformat.outputdir
----
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol closed the pull request at:
https://github.com/apache/incubator-predictionio/pull/436
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
GitHub user marevol reopened a pull request:
https://github.com/apache/incubator-predictionio/pull/436
[PIO-125] Add Spark 2.2 support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marevol/incubator-predictionio spark22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/436.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #436
----
commit 7ed1e9cb5282def385f49c0d01b8d4561d663bd4
Author: Shinsuke Sugaya <sh...@apache.org>
Date: 2017-09-20T10:47:28Z
add spark 2.2
commit a657cb50ca64998df3fe9ec01468421bd7dff67f
Author: Shinsuke Sugaya <sh...@apache.org>
Date: 2017-10-04T13:49:35Z
add mapreduce.output.fileoutputformat.outputdir
----
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
Re-run Travis.
I think it's better to reduce test matrix...
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol closed the pull request at:
https://github.com/apache/incubator-predictionio/pull/436
---
[GitHub] incubator-predictionio pull request #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol closed the pull request at:
https://github.com/apache/incubator-predictionio/pull/436
---
[GitHub] incubator-predictionio issue #436: [PIO-125] Add Spark 2.2 support
Posted by marevol <gi...@git.apache.org>.
Github user marevol commented on the issue:
https://github.com/apache/incubator-predictionio/pull/436
Oops, my travis result missed some results...
I'll fix this PR. This problem might come from HBase.
---