You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by avnerl <gi...@git.apache.org> on 2018/12/10 11:44:01 UTC
[GitHub] zeppelin pull request #3258: fix beam-runners-flink and zeppelin-scio scala ...
GitHub user avnerl opened a pull request:
https://github.com/apache/zeppelin/pull/3258
fix beam-runners-flink and zeppelin-scio scala version to 2.10
### What is this PR for?
build is failing when trying to use scala-2.11 profile
- beam-runners-flink 2.0.0 was never built with anything other than scala 2.10, therefore parametrizing its scala version breaks the build.
- scio is hard coded with scala 2.10, therefore parametrizing its scala version breaks the build.
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - merge or handle zeppelin-scio and beam-runner-flink scala versions
### What is the Jira issue?
* N/A
### How should this be tested?
* Pass CI
* manual build:
```
mvn clean package -Pbuild-distr -DskipTests -Pspark-2.4 -Phadoop-2.7 -Pscala-2.11
```
### Screenshots (if appropriate)
### Questions:
* no licenses files need update
* no breaking changes for older versions
* Does this needs documentation? not sure
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/avnerl/zeppelin master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3258.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3258
----
commit 28077ff39aaf18b8aaa8bb23c47fd4aa41a3b6b5
Author: avner <av...@...>
Date: 2018-12-10T07:53:09Z
fix beam-runners-flink and zeppelin-scio scala version to 2.10
----
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by avnerl <gi...@git.apache.org>.
Github user avnerl commented on the issue:
https://github.com/apache/zeppelin/pull/3258
> also we could upgrade beam interpreter to "require" beam 2.3 also (so it does build with scala-2.11)
this's what i did in: https://github.com/apache/zeppelin/pull/3258/commits/ffca03cbd4c9d6cb756eef1b21a1319f8d8c5759
do you have a better approach?
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/3258
in that case, can we make it build beam only when it has support for it.
ie. build scala 2.10 - beam 2.0.0
build scala 2.11 - beam not included (but not failing)
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by avnerl <gi...@git.apache.org>.
Github user avnerl commented on the issue:
https://github.com/apache/zeppelin/pull/3258
you are right. it didn't make sense. the reason i did this is because i wanted the build to pass. i remove the beam interpreter afterwards as i dint need it.
last 2 commits is reverting back to parameterized scala version in beam and introduced a new beam build profile that bump beam version to 2.3.0 when when building with `-Pscala-2.11`.
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by avnerl <gi...@git.apache.org>.
Github user avnerl commented on the issue:
https://github.com/apache/zeppelin/pull/3258
> > that's what i did in: [ffca03c](https://github.com/apache/zeppelin/commit/ffca03cbd4c9d6cb756eef1b21a1319f8d8c5759)
>
> ah, that would make different versions of beam with scala-2.10 or scala-2.11.
>
> is there a scio version that works with scala-2.11? what if we upgrade the supported version of beam and scio in all profile?
we don't have much choice here. beam was never crossed built with scala. starting version 2.3.0 it is built with 2.11 only (prior versions with 2.10).
so changing scala version implies changing beam version as well.
scio 0.2.4 is cross built with both 2.10 and 2.11, so it is already being taken care of.
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/3258
also we could upgrade beam interpreter to "require" beam 2.3 also (so it does build with scala-2.11)
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by avnerl <gi...@git.apache.org>.
Github user avnerl commented on the issue:
https://github.com/apache/zeppelin/pull/3258
do you think it's a good idea to tamper with beam version in `dev/change_scala_version.sh` too?
build will not pass if you do not use the `-Pscala-2.11` profile explicitly
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/3258
> that's what i did in: ffca03c
ah, that would make different versions of beam with scala-2.10 or scala-2.11.
is there a scio version that works with scala-2.11? what if we upgrade the supported version of beam and scio in all profile?
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by avnerl <gi...@git.apache.org>.
Github user avnerl commented on the issue:
https://github.com/apache/zeppelin/pull/3258
https://mvnrepository.com/artifact/org.apache.beam/beam-runners-flink
---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/3258
what about we don't build beam and scio if scala-2.11?
---