You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/03/16 09:37:46 UTC
[GitHub] [incubator-kyuubi] deadwind4 opened a new issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
deadwind4 opened a new issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
The following issue only achieves `FlinkSQLEngine` process, but users have no way to set `--jar` and `--library` options in Kyuubi. We should achieve this in `KyuubiServer` process too.
https://github.com/apache/incubator-kyuubi/issues/1685
https://github.com/SteNicholas/incubator-kyuubi/commit/dd5a704342334feb24eae981d184943f618dd228
We can put `--jar` and `--library` options in `kyuuubi-default.conf`, and KyuubiServer load these options.
### How should we improve?
`KyuubiServer` process loads the Flink `SQL Client startup options` in `kyuuubi-default.conf`. And then `KyuubiServer` transforms the options to standard Flink `SQL Client startup options` and passes them to `flink-sql-engine.sh` when the `flink-sql-engine.sh` process will be started.
The `kyuuubi-default.conf `options like the following.
```
flink.client.option.jar foo.jar
flink.client.option.library /foo/bar
```
Now `flink-sql-engine.sh` startup options are nothing, so users can't set any options.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1073016029
https://github.com/apache/incubator-kyuubi/blob/3eb1fa9e48b4c533f05b3719486df198c6e29751/externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala#L79
BTW, I think we can modify this line to support overwriting Flink configuration by `flink-sql-engine.sh`, then we can achieve same experience like Spark engine, which means we can define Flink confs in JDBC URL, `kyuubi-defaults.conf`, `flink-conf.yaml`, and the former has higher priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] SteNicholas removed a comment on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
SteNicholas removed a comment on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1068949811
@deadwind4, IMO, you could support all the start options of the Flink SQL client, not only `--jar` and `--library`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1073010939
> The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So `flink-sql-engine.sh` can not use configuration of Session or ExecuteStatement.
From [Flink document](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/jar/), Flink support `ADD JAR` and `REMOVE JAR` commands, what's difference between use `ADD JAR` and inject jars on launching Flink engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] deadwind4 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
deadwind4 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1086597527
https://issues.apache.org/jira/browse/FLINK-27010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] SteNicholas commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1068949811
@deadwind4, IMO, you could support all the start options of the Flink SQL client, not only `--jar` and `--library`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1073015017
> The `kyuuubi-default.conf `options like the following.
> ```
> flink.client.option.jar foo.jar
> flink.client.option.library /foo/bar
> ```
Give some comments based on my little understand of Flink, please correct me if I'm wrong.
There is no equality configuration in Flink of `--jar` and `--library` options defined by Flink CLI, if yes we'd better to introduce Kyuubi's style configurations instead of Flink's style to avoid potential conflict, e.g.
```
kyuubi.session.engine.flink.jar
kyuubi.session.engine.flink.library
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1073010939
> The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So `flink-sql-engine.sh` can not use configuration of Session or ExecuteStatement.
From [Flink document](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/jar/), Flink support `ADD JAR` and `REMOVE JAR` commands, what's the difference between use `ADD JAR` and inject jars on launching Flink engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] deadwind4 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
deadwind4 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1076312655
> > The `kyuuubi-default.conf `options like the following.
> > ```
> > flink.client.option.jar foo.jar
> > flink.client.option.library /foo/bar
> > ```
>
> Give some comments based on my little understand of Flink, please correct me if I'm wrong.
>
> There is no equality configuration in Flink of `--jar` and `--library` options defined by Flink CLI, if yes we'd better to introduce Kyuubi's style configurations instead of Flink's style to avoid potential conflict, e.g.
>
> ```
> kyuubi.session.engine.flink.jar
> kyuubi.session.engine.flink.library
> ```
I second with introducing Kyuubi's style configurations, but this has nothing to do with session.
Could we design prefix is `kyuubi.engine.flink.`?
```
kyuubi.engine.flink.jar
kyuubi.engine.flink.library
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] deadwind4 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
deadwind4 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1068993089
The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So `flink-sql-engine.sh` can not use configuration of Session or ExecuteStatement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] deadwind4 commented on issue #2153: [SUB-TASK][KPIP-2] Support Flink SQL Client startup options.
Posted by GitBox <gi...@apache.org>.
deadwind4 commented on issue #2153:
URL: https://github.com/apache/incubator-kyuubi/issues/2153#issuecomment-1076281964
> https://github.com/apache/incubator-kyuubi/blob/3eb1fa9e48b4c533f05b3719486df198c6e29751/externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala#L79
>
> BTW, I think we can modify this line to support overwriting Flink configuration by `flink-sql-engine.sh`, then we can achieve same experience like Spark engine, which means we can define Flink confs in JDBC URL, `kyuubi-defaults.conf`, `flink-conf.yaml`, and the former has higher priority.
I spilled a new issue, I will achieve this first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org