You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2022/11/06 14:02:54 UTC
[GitHub] [zeppelin] zjffdu opened a new pull request, #4506: [ZEPPELIN-5844] Support flink 1.16
zjffdu opened a new pull request, #4506:
URL: https://github.com/apache/zeppelin/pull/4506
### What is this PR for?
Introduce new module `flink1.16-shims` to support flink 1.16. Because the api incompatability of flink 1.16, abstract class FlinkShims changed some interfaces.
### What type of PR is it?
Feature
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5844
### How should this be tested?
* Manually tested on flink 1.16
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need to update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a diff in pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "Reamer (via GitHub)" <gi...@apache.org>.
Reamer commented on code in PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#discussion_r1086434414
##########
.github/workflows/core.yml:
##########
@@ -277,10 +277,10 @@ jobs:
auto-activate-base: false
use-mamba: true
- name: run tests for flink before 1.15 (exclusive)
- if: matrix.flink != '115'
- run: ./mvnw verify -pl flink/flink-scala-2.11,flink/flink-scala-2.12,flink-cmd,zeppelin-interpreter-integration -am -Pflink-${{ matrix.flink }} -Phadoop2 -Pintegration -DfailIfNoTests=false -Dtest=org.apache.zeppelin.flink.*Test,FlinkIntegrationTest${{ matrix.flink }} ${MAVEN_ARGS}
- - name: run tests for flink before 1.15 (inclusive)
- if: matrix.flink == '115'
+ if: matrix.flink < '115'
+ run: ./mvnw verify -pl flink/flink-scala-2.11,flink/flink-scala-2.12,flink-cmd,zeppelin-interpreter-integration -Pflink-${{ matrix.flink }} -Phadoop2 -Pintegration -DfailIfNoTests=false -B -Dtest=org.apache.zeppelin.flink.*Test,FlinkIntegrationTest${{ matrix.flink }} ${MAVEN_ARGS}
Review Comment:
`-B` is part of `${MAVEN_ARGS} it is therefore not necessary at this point.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "Reamer (via GitHub)" <gi...@apache.org>.
Reamer commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1410210938
I think we can merge this change and move forward with #4558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1311634940
> > Will merge if no more comments
>
> No automated tests?
Oops, missing in the commit, let me add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu merged pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "zjffdu (via GitHub)" <gi...@apache.org>.
zjffdu merged PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] canhtran commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "canhtran (via GitHub)" <gi...@apache.org>.
canhtran commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1428335587
@zjffdu Hi Zeff, thanks for the reply
I followed the link here https://zeppelin.apache.org/docs/latest/setup/basics/how_to_build.html.
1. Clone the latest source of zeppelin
2. `mvn clean package -DskipTests`
I had another Zeppelin source with the commit at `deb7f3b1c4c4a1b82ef85e59a327f7c8ad6f4714`, It does have the Flink interpreter
<img width="696" alt="image" src="https://user-images.githubusercontent.com/3214379/218525877-39e3bbb7-2487-4c8c-9ec6-ecae7477aec2.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] PrabhuJoseph commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
PrabhuJoseph commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1356800918
Nevermind, pyflink worked after copying /usr/lib/flink/opt/flink-python-1.16.0.jar to /usr/lib/flink/lib.
Except the regression of [FLINK-13645](https://issues.apache.org/jira/browse/FLINK-13645), this patch works perfectly for us. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1357122735
Thanks for the feedback @PrabhuJoseph @zenozeng @vineethNaroju , I am busy on other stuff recently, will take a look at this later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1313401152
> BTW, a test related to this change failed. Is it fine? https://github.com/apache/zeppelin/actions/runs/3448729205/jobs/5755992907
I will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
jongyoul commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1313399654
BTW, a test related to this change failed. Is it fine? https://github.com/apache/zeppelin/actions/runs/3448729205/jobs/5755992907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] canhtran commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "canhtran (via GitHub)" <gi...@apache.org>.
canhtran commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1451342014
Thanks @PrabhuJoseph. It works by copying the `flink-sql-client-1.16.jar` to `lib` folder. The documentation missed this step, I will raise a PR for it. Thanks again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a diff in pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "zjffdu (via GitHub)" <gi...@apache.org>.
zjffdu commented on code in PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#discussion_r1086503324
##########
.github/workflows/core.yml:
##########
@@ -277,10 +277,10 @@ jobs:
auto-activate-base: false
use-mamba: true
- name: run tests for flink before 1.15 (exclusive)
- if: matrix.flink != '115'
- run: ./mvnw verify -pl flink/flink-scala-2.11,flink/flink-scala-2.12,flink-cmd,zeppelin-interpreter-integration -am -Pflink-${{ matrix.flink }} -Phadoop2 -Pintegration -DfailIfNoTests=false -Dtest=org.apache.zeppelin.flink.*Test,FlinkIntegrationTest${{ matrix.flink }} ${MAVEN_ARGS}
- - name: run tests for flink before 1.15 (inclusive)
- if: matrix.flink == '115'
+ if: matrix.flink < '115'
+ run: ./mvnw verify -pl flink/flink-scala-2.11,flink/flink-scala-2.12,flink-cmd,zeppelin-interpreter-integration -Pflink-${{ matrix.flink }} -Phadoop2 -Pintegration -DfailIfNoTests=false -B -Dtest=org.apache.zeppelin.flink.*Test,FlinkIntegrationTest${{ matrix.flink }} ${MAVEN_ARGS}
Review Comment:
Thanks @Reamer , I have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] PrabhuJoseph commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
PrabhuJoseph commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1350801311
@zjffdu @zenozeng Thanks for this patch, this is very useful to us.
We are facing [FLINK-13645](https://issues.apache.org/jira/browse/FLINK-13645) again after Flink-1.16 upgrade. It is working fine on Flink-1.15. Looks [FLINK-15635](https://issues.apache.org/jira/browse/FLINK-15635) is causing the regression.
Do you know how to fix this issue. I have used FLINK-13645 fix which picks java.net.URLClassLoader but does not help.
**Repro: Flink1.16 (runs on YARN) + With this Zeppelin Patch**
```
%flink.bsql
DROP TABLE IF EXISTS bank_raw;
CREATE TABLE bank_raw (
content STRING
) WITH (
'format.field-delimiter'='\n',
'connector.type'='filesystem',
'format.derive-schema'='true',
'connector.path'='/tmp/bank.csv',
'format.type'='csv'
);
DROP TABLE IF EXISTS bank;
CREATE TABLE bank (
age int,
job string,
marital string,
education string,
`default` string,
balance string,
housing string,
loan string,
contact string,
`day` string,
`month` string,
duration int,
campaign int,
pdays int,
previous int,
poutcome string,
y string
) WITH (
'format.field-delimiter'=',',
'connector.type'='filesystem',
'format.derive-schema'='true',
'connector.path'='/tmp/bank_cleaned',
'format.type'='csv'
);
%flink
import org.apache.flink.api.java.typeutils.RowTypeInfo
import org.apache.flink.api.common.typeinfo.Types
import org.apache.flink.api.java.typeutils._
import org.apache.flink.api.scala.typeutils._
import org.apache.flink.api.scala._
class Person(val age:Int, val job: String, val marital: String, val education: String, val default: String, val balance: String, val housing: String, val loan: String, val contact: String, val day: String, val month: String, val duration: Int, val campaign: Int, val pdays: Int, val previous: Int, val poutcome: String, val y: String)
class ParseFunction extends TableFunction[Row] {
def eval(line: String) {
val tokens = line.split(";")
// parse the line
if (!line.startsWith("\"age\"")) {
collect(Row.of(new Integer(tokens(0).toInt), normalize(tokens(1)), normalize(tokens(2)), normalize(tokens(3)), normalize(tokens(4)), normalize(tokens(5)), normalize(tokens(6)), normalize(tokens(7)), normalize(tokens(8)), normalize(tokens(9)), normalize(tokens(10)), new Integer(tokens(11).toInt), new Integer(tokens(12).toInt),
new Integer(tokens(13).toInt), new Integer(tokens(14).toInt), normalize(tokens(15)), normalize(tokens(16))))
}
}
override def getResultType() = {
val cls = classOf[Person]
Types.ROW(Types.INT, Types.STRING, Types.STRING, Types.STRING,Types.STRING,Types.STRING,Types.STRING,Types.STRING,Types.STRING,Types.STRING,Types.STRING,
Types.INT, Types.INT, Types.INT, Types.INT, Types.STRING, Types.STRING)
}
// remove the quote
private def normalize(token: String) = {
if (token.startsWith("\"")) {
token.substring(1, token.length - 1)
} else {
token
}
}
}
btenv.registerFunction("parse", new ParseFunction())
%flink.bsql
insert into bank select T.* from bank_raw, LATERAL TABLE(parse(content)) as T(age, job, marital, education, `default`, balance, housing, loan, contact, `day`, `month`, duration, campaign, pdays, previous, poutcome, y)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zenozeng commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
zenozeng commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1343483428
Looking forward to this patch +1.
BTW, I have manually tested this PR on flink 1.16, it worked as expected in all my use cases. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] halfvector commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
halfvector commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1343035090
Looking forward to this patch, thank you for starting it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] canhtran commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "canhtran (via GitHub)" <gi...@apache.org>.
canhtran commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1432832112
It seems like the issue starting from #4558
I built the zeppelin at commit `f30ea77accc921db3f06aadd5675f71140bc6da2` and it's fine
<img width="647" alt="image" src="https://user-images.githubusercontent.com/3214379/219334347-08ed0407-fdc6-4f1e-99da-15cbefa7c5ed.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "zjffdu (via GitHub)" <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1427386135
@canhtran How did you build Zeppelin?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "zjffdu (via GitHub)" <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1402059634
@halfvector @zenozeng @PrabhuJoseph @vineethNaroju I have updated this PR and CI is passed now. Feel free to try it to see whether it solves your problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
Reamer commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1311633222
> Will merge if no more comments
No automated tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "zjffdu (via GitHub)" <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1432962218
Thanks @canhtran for the feedback, https://github.com/apache/zeppelin/pull/4558 indeed cause this issue. The workaround is using `mvn clean package -DskipTests -Pflink-113`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] canhtran commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "canhtran (via GitHub)" <gi...@apache.org>.
canhtran commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1451248958
I'm also facing the below issue while testing the zeppelin with flink 1.16. I guess we need to modify/copy the libraries in flink to resolve it but haven't figured it out.
```
org.apache.zeppelin.interpreter.InterpreterException: java.lang.NoClassDefFoundError: org/apache/flink/table/client/util/ClientWrapperClassLoader
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/client/util/ClientWrapperClassLoader
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.zeppelin.flink.FlinkShims.loadShims(FlinkShims.java:68)
at org.apache.zeppelin.flink.FlinkShims.getInstance(FlinkShims.java:86)
at org.apache.zeppelin.flink.FlinkScalaInterpreter.initFlinkConfig(FlinkScalaInterpreter.scala:158)
at org.apache.zeppelin.flink.FlinkScalaInterpreter.open(FlinkScalaInterpreter.scala:122)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:71)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.table.client.util.ClientWrapperClassLoader
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 16 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
zjffdu commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1311620071
Will merge if no more comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] ebongzzang commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "ebongzzang (via GitHub)" <gi...@apache.org>.
ebongzzang commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1570008374
I've encountered below exception setup flink 1.16.0 with zeppelin interpreter. (mode = remote)
`NoClassDefFoundError: org/apache/flink/table/api/bridge/scala/internal/StreamTableEnvironmentImpl`
After downloading jar `flink-table-api-scala-bridge_2.12` from maven and moving under `flink/lib`, exception was cleared and Flink interpreter works normally.
Am I just a mistake when building zeppelin? here is executed build command.
`mvn clean package -Pspark-3.2 -Ppyspark -Pr -Pscala-2.11 -Pbuild-distr -DskipTests -Pflink-116`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] PrabhuJoseph commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
PrabhuJoseph commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1356781725
@zjffdu @zenozeng We are facing below issue when running a pyflink from Zeppelin with this patch on Flink-1.16. It is working fine on Flink-1.15. Any idea on what could be the issue.
```
%flink.pyflink
table = st_env.sql_query("select * from bank limit 10")
z.show(table)
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to bootstrap pyflink
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:844)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:752)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to bootstrap pyflink
at org.apache.zeppelin.flink.PyFlinkInterpreter.open(PyFlinkInterpreter.java:97)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
... 8 more
Caused by: java.io.IOException: Fail to run bootstrap script: python/zeppelin_pyflink.py
Fail to execute line 44: st_env = StreamTableEnvironment(intp.getJavaStreamTableEnvironment())
Traceback (most recent call last):
File "/tmp/zeppelin_python.py", line 180, in <module>
exec(code, _zcUserQueryNameSpace)
File "<stdin>", line 44, in <module>
File "/usr/lib/flink/opt/python/pyflink.zip/pyflink/table/table_environment.py", line 1653, in __init__
super(StreamTableEnvironment, self).__init__(j_tenv)
File "/usr/lib/flink/opt/python/pyflink.zip/pyflink/table/table_environment.py", line 98, in __init__
self._set_python_executable_for_local_executor()
File "/usr/lib/flink/opt/python/pyflink.zip/pyflink/table/table_environment.py", line 1537, in _set_python_executable_for_local_executor
if not j_config.containsKey(jvm.PythonOptions.PYTHON_EXECUTABLE.key()) \
TypeError: 'JavaPackage' object is not callable
at org.apache.zeppelin.python.PythonInterpreter.bootstrapInterpreter(PythonInterpreter.java:580)
at org.apache.zeppelin.flink.PyFlinkInterpreter.open(PyFlinkInterpreter.java:95)
... 9 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] canhtran commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "canhtran (via GitHub)" <gi...@apache.org>.
canhtran commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1427370649
I'm rebuilding from source for flink 1.16.0 but there aren't any `Flink interpreter` after the build. There is only `flink-cmd interpreter`. Please let me know if there is anything that I'm missing for this version of Zeppelin and Flink? Thanks in advance for your advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] bigjar commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "bigjar (via GitHub)" <gi...@apache.org>.
bigjar commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1451139614
Hi, @zjffdu. when I use flink.udf.jars, I found two problems. One is it can not find the org.apache.zeppelin.shaded.com.google.common.io.Files, because the #4545 had change the shaded prefix. Another is the variable userJars should be transfrom to the file before transform to the URL when it used in createTableEnvs functions in the FlinkScalaInterpreter line 434.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] PrabhuJoseph commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by "PrabhuJoseph (via GitHub)" <gi...@apache.org>.
PrabhuJoseph commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1451314800
> I'm also facing the below issue while testing the zeppelin with flink 1.16. I guess we need to modify/copy the libraries in flink to resolve it but haven't figured it out.
I think this happens as Flink Sql Client Jar is not part of Zeppelin Flink Interpreter Classpath. Could you test after copying /usr/lib/flink/opt/flink-sql-client-1.16.jar into either Flink Lib /usr/lib/flink/lib or Zeppelin Flink Interpreter Classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zenozeng commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
zenozeng commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1344397391
@vineethNaroju As the Flink documentation says, you can move `flink-sql-client*.jar` to the /lib folder.
> The /lib directory of the Flink distribution additionally contains various JARs including commonly used modules, such as all the required [modules to execute Table jobs](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/advanced/#anatomy-of-table-dependencies) and a set of connector and formats. These are loaded by default and can be removed from the classpath just by removing them from the /lib folder.
> Flink also ships additional optional dependencies under the /opt folder, which can be enabled by moving the JARs in the /lib folder.
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/advanced/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] vineethNaroju commented on pull request #4506: [ZEPPELIN-5844] Support flink 1.16
Posted by GitBox <gi...@apache.org>.
vineethNaroju commented on PR #4506:
URL: https://github.com/apache/zeppelin/pull/4506#issuecomment-1344277755
Thank you Jeff for this patch. The patch worked except one issue - with flink interpreter failed to start with below exception as,
```
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/client/util/ClientWrapperClassLoader
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.zeppelin.flink.FlinkShims.loadShims(FlinkShims.java:71)
at org.apache.zeppelin.flink.FlinkShims.getInstance(FlinkShims.java:89)
at org.apache.zeppelin.flink.FlinkScalaInterpreter.initFlinkConfig(FlinkScalaInterpreter.scala:158)
at org.apache.zeppelin.flink.FlinkScalaInterpreter.open(FlinkScalaInterpreter.scala:122)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:71)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
```
The zeppelin's interpreter.sh includes $FLINK_HOME/lib into the flink interpreter class path but it also requires $FLINK_HOME/opt/flink-sql-client*.jar . The below changes in interpreter.sh helped to fix the issue.
```
FLINK_SQL_CLIENT_JAR=$(find "${FLINK_HOME}/opt" -name 'flink-sql-client*.jar')
ZEPPELIN_INTP_CLASSPATH+=":${FLINK_PYTHON_JAR}:${FLINK_APP_JAR}:${FLINK_SQL_CLIENT_JAR}"
```
Could you please include this also if it helps ? Please let us know if something is missing from our setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org