You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2023/01/16 09:05:25 UTC
[GitHub] [kyuubi] ulysses-you opened a new issue, #4175: [Subtask] Docker image tag options
ulysses-you opened a new issue, #4175:
URL: https://github.com/apache/kyuubi/issues/4175
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues) and found no similar issues.
### Describe the subtask
Kyuubi supports multi-engines, e.g. Spark, Flink, Hive... So there is a chance to provide some custom image tags since we are using a separate repo.
An idea of tag options:
version | tag | derived tags
-- | -- | --
1.6.0 Kyuubi only | 1.6.0-scala_2.12-java8-ubuntu | 1.6.0, latest
1.6.0 with Spark default version binary tar | 1.6.0-spark3.3-scala_2.12-java8-ubuntu | 1.6.0-spark3.3, 1.6.0-spark
1.6.0 with Flink default version binary tar | 1.6.0-flink1.5-scala_2.12-java8-ubuntu | 1.6.0-flink3.3, 1.6.0-flink
1.6.0 with all default version binary tar | 1.6.0-all-scala_2.12-java8-ubuntu | 1.6.0-all
### Parent issue
https://github.com/apache/kyuubi/issues/4158
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384227358
And as Kyuubi contains extensions for engine, consider release an `all-in-one` tag version shipping the possible available extensions and configs for best practice or proficiency as well.
e.g. including,
- Auxiliary Optimization with enabling configs (https://kyuubi.readthedocs.io/en/master/extensions/engines/spark/rules.html)
- Authz Plugin for Spark without enabling (https://kyuubi.readthedocs.io/en/master/security/authorization/spark/index.html)
- SQL Lineage Support without enabling (https://kyuubi.readthedocs.io/en/master/extensions/engines/spark/lineage.html)
- Hive Dialect Support without enabling (https://kyuubi.readthedocs.io/en/master/extensions/engines/spark/jdbc-dialect.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you commented on issue #4175: [Subtask] Docker image tag options
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1383763582
yea, `scala_2.12-java8-ubuntu` is more like a style of DOI. For `apache/kyuubi` repo, we can use the derived tag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384690761
spark official docker image is also using ubuntu, https://github.com/apache/spark-docker/tree/master/3.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384724457
Yes, thanks for sharing this status in your company. With all the discussions, `ubuntu` is a balanced choice for the base image. Even for the image size, it is not as extremely small as alpine but already smaller than centos.
`ubuntu` is still the first option for this topic, since it is adopted in Spark's official image (as you just pointed out), but also in Flink's (https://github.com/apache/flink-docker).
Meanwhile, it's not good enough for the situation in my case if this's the only option for base os, and it's pushing me away from practise and having more thoughts in reassembling the docker images in private distribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zwangsheng commented on issue #4175: [Subtask] Docker image tag options
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1383723856
+1 for provided `basic`, `spark`, `flink` and `all` images.
For tag, maybe we should omit `scala_2.12-java8-ubuntu` as not provided another options.
We can indicate those versions in the description, so that users can perceive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384681791
> As for the choice of operation system, I would prefer options in either `alpine` or `centos`.
Ubuntu looks fine for me, all the image in ebay is based on ubuntu or migrating to ubuntu.
-1 for centos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384718530
> never heard of using ubuntu in production either for server deployment or container images in the infrastructure industry (e.g. finance, power grid and etc.).
our old hadoop nodes images were based on centos before, but now, all the hadoop nodes are based on ubuntu or in the migration.
I think If I did not say that, you will not know that we are using ubuntu.
There might be some limitations for my view, because I just know the current status in ebay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384221133
As for the choice of operation system, I would prefer options in either `alpine` or `centos`.
Considering the maturity in industry production and package management,
- `ubuntu` is less used as base docker image in production
- `centos` is more common in production in practise , but it's no longer maintained from official
- `alpine` is popular and practically used as small-size Linux base images in Docker, generally used from app to service deployment. plus, the `apk` is easy to use and includes most common package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you closed issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you closed issue #4175: [Subtask][DISCUSS] Docker image tag options
URL: https://github.com/apache/kyuubi/issues/4175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you commented on issue #4175: [Subtask] Docker image tag options
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1383704229
This issue is used to disscuss what image tag should be pushed during each release. cc @yaooqinn @pan3793 @zwangsheng @hddong @yanghua @turboFei @cfmcgrady @cxzl25 and all. Any thought is welcome !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384243018
Also, to maximize both performance, accessibility, and potentiality in best practices and potentiality out of box, consider release an fine-tuned `all-in-one` image for each engine with engine/server plugins and suggested configs of by default.
eg.
- [engine connector] Spark Hive Connector (https://kyuubi.readthedocs.io/en/master/connector/spark/hive.html?highlight=hive%20connector#hive-connector-integration)
- [engine plugin] Auxiliary Optimization, enabled by default (https://kyuubi.readthedocs.io/en/master/extensions/engines/spark/rules.html)
- [engine plugin] Authz Plugin for Ranger, disabled by default
- [engine plugin] SQL Lineage Support, disabled by default
- [engine plugin] Hive Dialect Support, disabled by default (https://kyuubi.readthedocs.io/en/master/extensions/engines/spark/jdbc-dialect.html)
...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384707364
`centos` is just an option for discussion as many companies already have local proxy for packages , I am good with both `ubuntu` and `alpine`.
But considering in 2 facts,
- `ubuntu` is good for software builds with development tools, but does not contribute too in runtime. For container deployment , size matters. The compressed size of distribution will be up to 1G (and 1.5G decompressed), as 400MB OS lib with ubuntu + 250MB JDK + 250 MB Kyuubi with extensions. As with `alpine`, the whole size can be shrunk down to 600-700MB in total and 1G in decompressed.
- never heard of using `ubuntu` in production either for server deployment or container images in the financial or other infrastructure industry.
- from base image adoption rate, even debian has a greater share than ubuntu
I am quite sure the community will agree to ship a version based on ubuntu, but it is never the first choice for production for me and other users with similar background or consideration. And that's why I raise the discussion for a second option of OS base lib here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384726441
@bowenliang123 can you show some projects who ships centOS to image ? That may help us to see the difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384234182
As for the choice of operation system type of base images, I would prefer `alpine` or `centos`.
From a prospective of production maturity or package management
- `ubuntu` is less used in server or containers deployment in production, for Ops means different tool set and packages
- `centos` is more popular in the pass , but good enough for production . lacking of further official maintainance.
- `alpine` is one of most popular base images with minimized image size while providing comprehensive features. OpenJDK also ships official version with it in new versions although alpine uses glibc. `apk` in `alpine` is easy to use and covering most common packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on issue #4175: [Subtask][DISCUSS] Docker image tag options
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on issue #4175:
URL: https://github.com/apache/kyuubi/issues/4175#issuecomment-1384732277
`centos` is no longer supported and we could skip this option for kyuubi.
And my concern is for enterprise-level production use,
- support `yum` like package management and `rpm` installers
- make use of the existing mirror proxy or private repo of the company, which is the key to align for Ops in deployment or management for both raw machines or containers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org