You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lenglet-k (via GitHub)" <gi...@apache.org> on 2025/02/18 16:57:58 UTC
[I] lighten docker images [pulsar-helm-chart]
lenglet-k opened a new issue, #571:
URL: https://github.com/apache/pulsar-helm-chart/issues/571
**Is your feature request related to a problem? Please describe.**
It's not a problem but just a question, why use image apachepulsar/pulsar:tag and not apachepulsar/pulsar-all:tag ?
Indeed the size of apachepulsar/pulsar-all:tag it's to 3gb and apachepulsar/pulsar:tag it's 446Mb ? What is in terms of storage and downloading would be more interesting.
I imagine that some binary are missing in the lightened image. But I see that there is also a 2gb /pulsar/connectors directory in the apachepulsar/pulsar-all:tag image.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] lighten docker images [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #571:
URL: https://github.com/apache/pulsar-helm-chart/issues/571#issuecomment-2668011332
> Ok thanks for your feedback. this means I can use the apachepulsar/pulsar image to deploy pulsar and therefore use the lightest image. I was thinking there might be scripts or something used in the pulsar-all image to instantiate the cluster or do other configurations.
In most cases there won't be much benefit of using apachepulsar/pulsar if pulsar-all gets used eventually since a k8s node will cache the pulled images. Obviously there will be some benefit when the initial jobs start faster, without downloading the full pulsar-all image. We should target eliminating the use of pulsar-all image.
Even when using Pulsar Functions and Pulsar IO connectors, most users could upload the connectors they use separately. It's also possible to make the cluster download from an external URL as long as the URL is whitelisted in the configuration (`additionalEnabledConnectorUrlPatterns` and `additionalEnabledFunctionsUrlPatterns` in `conf/functions_worker.yml`).
> I can also propose an apachepulsar/pulsar-all image without adding the connectors.
For legacy reasons, I think a new name should be picked. I guess most users would only need tiered-storage ("offloaders") in the image for a Pulsar cluster in addition to the plain Pulsar image (which should be kept as small as possible).
We could start publishing a new image with tiered-storage if that would be sufficient.
Perhaps the name could be `apachepulsar/pulsar-cluster` so that we don't limit it only for adding `tiered-storage` into the image.
I checked the tiered-storage modules and they are very large due to duplicating unnecessary dependencies inside the nar files. A similar solution as https://github.com/apache/pulsar/pull/9638 should be applied to the tiered-storage nar files (using `provided` scope for dependencies available in Pulsar).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] lighten docker images [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #571:
URL: https://github.com/apache/pulsar-helm-chart/issues/571#issuecomment-2721008364
Here's an example how to build a custom image on top of apachepulsar/pulsar with needed connectors:
https://github.com/apache/pulsar/discussions/24075#discussioncomment-12487275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] lighten docker images [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #571:
URL: https://github.com/apache/pulsar-helm-chart/issues/571#issuecomment-2666712157
The Pulsar IO connectors could also be size optimized, but that's some work that never happened.
A long time ago I reported https://github.com/apache/pulsar/issues/9572 and made an attempt in https://github.com/apache/pulsar/pull/9638 to address the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] lighten docker images [pulsar-helm-chart]
Posted by "lenglet-k (via GitHub)" <gi...@apache.org>.
lenglet-k commented on issue #571:
URL: https://github.com/apache/pulsar-helm-chart/issues/571#issuecomment-2667753546
Ok thanks for your feedback. this means I can use the apachepulsar/pulsar image to deploy pulsar and therefore use the lightest image. I was thinking there might be scripts or something used in the pulsar-all image to instantiate the cluster or do other configurations.
I can also propose an apachepulsar/pulsar-all image without adding the connectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] lighten docker images [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #571:
URL: https://github.com/apache/pulsar-helm-chart/issues/571#issuecomment-2666697117
Thanks for bringing this up, @lenglet-k. There's isn't a useful reason to use apachepulsar/pulsar-all when Pulsar IO Connectors aren't used or tiered storage ("offloaders") isn't used.
I agree that we should make a switch to use apachepulsar/pulsar by default. It's just due to legacy reasons that we have kept on going ahead using apachepulsar/pulsar-all as the default.
These are the Apache Pulsar branch-4.0 Dockerfiles:
* [Dockerfile for pulsar-all](https://github.com/apache/pulsar/blob/branch-4.0/docker/pulsar-all/Dockerfile)
* [Dockerfile for pulsar](https://github.com/apache/pulsar/blob/branch-4.0/docker/pulsar/Dockerfile)
Contributions are welcome to improve the current situation. Once we make the switch, we could bump the Pulsar helm chart version to 4.0.0 to indicate a major change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org