You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/21 12:43:19 UTC
[GitHub] [pulsar-helm-chart] lhotari opened a new issue #110: Pulsar proxy fails to start with pulsar Docker image that uses non-root user
lhotari opened a new issue #110:
URL: https://github.com/apache/pulsar-helm-chart/issues/110
### Problem
There's a permission issue in pulsar-proxy when using a Docker image that uses a non-root user.
The 2.8.0-SNAPSHOT Pulsar docker images use a non-root user. This change was made in https://github.com/apache/pulsar/pull/8796 .
Since the default configuration uses port 80 and now when the default user is "pulsar", it cannot bind to port 80.
This is the error log in pulsar-proxy-0 pod:
```
14:38:21.477 [main] ERROR org.apache.pulsar.proxy.server.ProxyServiceStarter - Failed to start pulsar proxy service. error msg Failed to start HTTP server on ports [80]
java.io.IOException: Failed to start HTTP server on ports [80]
at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:246) ~[org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.proxy.server.ProxyServiceStarter.<init>(ProxyServiceStarter.java:177) [org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.proxy.server.ProxyServiceStarter.main(ProxyServiceStarter.java:186) [org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
Caused by: java.net.SocketException: Permission denied
at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_282]
at sun.nio.ch.Net.bind(Net.java:461) ~[?:1.8.0_282]
at sun.nio.ch.Net.bind(Net.java:453) ~[?:1.8.0_282]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) ~[?:1.8.0_282]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) ~[?:1.8.0_282]
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:345) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[org.eclipse.jetty-jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.server.Server.doStart(Server.java:401) ~[org.eclipse.jetty-jetty-server-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[org.eclipse.jetty-jetty-util-9.4.35.v20201120.jar:9.4.35.v20201120]
at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:224) ~[org.apache.pulsar-pulsar-proxy-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
... 2 more
2021-03-19 02:38:21,480 [sun.misc.Launcher$AppClassLoader@7ef20235] error Uncaught exception in thread main: java.io.IOException: Failed to start HTTP server on ports [80]
```
The problem goes away after changing the port to 8080 in values.yaml:
```
proxy:
ports:
http: 8080
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] cdbartholomew commented on issue #110: Pulsar proxy fails to start with pulsar Docker image that uses non-root user
Posted by GitBox <gi...@apache.org>.
cdbartholomew commented on issue #110:
URL: https://github.com/apache/pulsar-helm-chart/issues/110#issuecomment-846198394
@michaeljmarshall Helm has a built-in variable that can tell you whether it is doing an install or an upgrade. For example, you can do something like this:
```
{{- if .Release.IsInstall }}
do something
{{ end }}
```
Alternatively, you can use [Helm Hooks](https://helm.sh/docs/topics/charts_hooks/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #110: Pulsar proxy fails to start with pulsar Docker image that uses non-root user
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #110:
URL: https://github.com/apache/pulsar-helm-chart/issues/110#issuecomment-845501809
@sijie and @lhotari - what are your thoughts here? It'd be good to solve this soon so that it doesn't affect end users when we release 2.8.0 soon.
I see a couple options.
1. We could add a value for the `targetPort` on the pulsar proxy service where the `targetPort` defaults to `8080`. In this case, we could update the template for the proxy service with a block like this:
```yaml
spec:
type: {{ .Values.proxy.service.type }}
ports:
{{- if or (not .Values.tls.enabled) (not .Values.tls.proxy.enabled) }}
- name: http
port: {{ .Values.proxy.ports.http }}
targetPort: {{ .Values.proxy.ports.targetHttp }}
protocol: TCP
```
Then, we would open up the `targetHttp` port on the proxy container and pod definition. This would be helpful because it'd be a seamless transition for end users targeting the service. Technically, any users hitting the pods directly would be affected by this breaking change. Although, I believe that kind of direct interations is an anti-pattern within kubernetes, so I think we could safely ignore that case. Further, the proxy's `PodMonitor` config is set to hit the named port `http`, which would mean that it'd make the transition to an arbitrary `targetPort` seamlessly. (I technically still need to double check that this whole solution works. I'm pretty confident it will, but I'll need to test before pushing up the PR.)
A major benefit to this solution is that it lets us support rootless containers by default for our end users.
1. As I suggested in my earlier comment, we could default the Pulsar Proxy container to run as the root user, and then let end users manage the upgrade themselves. This solution seems more complicated to me, and it doesn't result in a great experience for end users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #110: Pulsar proxy fails to start with pulsar Docker image that uses non-root user
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #110:
URL: https://github.com/apache/pulsar-helm-chart/issues/110#issuecomment-846169762
In thinking more about this more, we have a problem for our users that upgrade to 2.8.0 from any previous version of pulsar. The persistent data written by prior bookie and zookeeper pods will be owned by the root user, not the new `pulsar` user, and it's possible that directories like the `data` directory in the bookie will not be writable for the `pulsar` user. Given that zookeepers and bookies delete files, they will need more permission in order to function properly after upgrade.
Solution 1 (mentioned above) is not sufficient to prevent breaking clusters during upgrade. We need to either run bookie and zookeeper containers as the root user by default and have users opt in to running them as non-root containers, or we need to solve the upgrade problem in the chart. Perhaps we can solve this by providing an optional init container that properly changes file system ownership for the bookie/zookeeper pods. Given that our users have already been running the containers as the root user, I don't see any conflict in using a privileged init container just this once.
Whatever path we choose, I would like to try to give new users a rootless deployment by default. I am not familiar with how helm manages upgrades versus new deployments, but perhaps there is a way to have conditional logic that branches on upgrade vs new deployment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #110: Pulsar proxy fails to start with pulsar Docker image that uses non-root user
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #110:
URL: https://github.com/apache/pulsar-helm-chart/issues/110#issuecomment-803793674
I proposed the following solution in https://github.com/apache/pulsar/pull/8796#issuecomment-802923466. It's copied here:
> I think the easiest way to prevent a breaking change in the helm chart is to default to using the `root` user in the helm chart's deployment for the pulsar proxy. We could then add a configuration that allows users to opt in to running that container as a non root user.
>
> Essentially, we need to default a pulsar proxy pod's security context to the following:
>
> ```yaml
> securityContext:
> runAsUser: 0
> runAsGroup: 0
> ```
>
> I believe all other pulsar containers should still be able to run as the non root user.
>
> (I am assuming that we want to prevent breaking changes here as the 2.8.0 release is only a minor version bump.)
I'm happy to discuss alternatives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org