You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/28 17:22:47 UTC
[GitHub] [pulsar] ckdarby opened a new issue #7089: [Helm] Proxy started throwing netty errors even on a new cluster
ckdarby opened a new issue #7089:
URL: https://github.com/apache/pulsar/issues/7089
**Describe the bug**
When starting up a new cluster we're getting errors from the proxy related to netty thread allocation problems but I'm confused as the docker image is still using the tagged 2.5.0 and I don't see any pulled in external libs.
**To Reproduce**
Steps to reproduce the behaviour:
1. Helm install
2. k logs dga-detection-pulsar-proxy-0
**Expected behavior**
Proxy to come up as expected
**Screenshots**
k logs with the stacktrace, [netty.txt](https://github.com/apache/pulsar/files/4697235/netty.txt).
**Desktop (please complete the following information):**
- Kubernetes 1.16
- v3.1.1
- Using helm chart from apache/pulsar-helm-chart
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ckdarby commented on issue #7089: [Helm] Proxy started throwing netty errors even on a new cluster
Posted by GitBox <gi...@apache.org>.
ckdarby commented on issue #7089:
URL: https://github.com/apache/pulsar/issues/7089#issuecomment-635565265
So, we rolled back everything our side, helm charts back to the known version, nuked the whole k8s cluster, made a new K8s cluster, etc etc. Eventually the only difference we had was a node group had been changed from r5.large to r5.4xlarge to test out #7058
Looking at Netty docs @nickelozz found
> The number of threads you need is determined by the components within Jetty you start to use and the hardware present on your system (the number of cpu cores and network interfaces impacts nio and consequently your thread requirements).
He was also nice enough to find this from AWS docs:
I looked through the Pulsar proxy code, [here](https://github.com/apache/pulsar/blob/master/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java#L466) and see that it doesn't factor in anything related to interfaces :sob:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ckdarby commented on issue #7089: [Helm] Proxy started throwing netty errors even on a new cluster
Posted by GitBox <gi...@apache.org>.
ckdarby commented on issue #7089:
URL: https://github.com/apache/pulsar/issues/7089#issuecomment-635615954
We changed the numThreads in the helm chart from 8->12 and the proxy came up.
We'll make a PR on the apache/pulsar-helm-chart to allow this to be exposed out in the values.yaml.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ckdarby closed issue #7089: [Helm] Proxy started throwing netty errors even on a new cluster
Posted by GitBox <gi...@apache.org>.
ckdarby closed issue #7089:
URL: https://github.com/apache/pulsar/issues/7089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7089: [Helm] Proxy started throwing Jetty errors even on a new cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7089:
URL: https://github.com/apache/pulsar/issues/7089#issuecomment-636239030
So you need to adjust the settings based on your machine type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7089: [Helm] Proxy started throwing Jetty errors even on a new cluster
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7089:
URL: https://github.com/apache/pulsar/issues/7089#issuecomment-636238611
@ckdarby This seems to be a known issue related to jetty. You need to adjust the number threads to be more than cpu cores.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ckdarby edited a comment on issue #7089: [Helm] Proxy started throwing Jetty errors even on a new cluster
Posted by GitBox <gi...@apache.org>.
ckdarby edited a comment on issue #7089:
URL: https://github.com/apache/pulsar/issues/7089#issuecomment-635565265
So, we rolled back everything our side, helm charts back to the known version, nuked the whole k8s cluster, made a new K8s cluster, etc etc. Eventually the only difference we had was a node group had been changed from r5.large to r5.4xlarge to test out #7058
Looking at jetty docs @nickelozz found
> The number of threads you need is determined by the components within Jetty you start to use and the hardware present on your system (the number of cpu cores and network interfaces impacts nio and consequently your thread requirements).
He was also nice enough to find this from AWS docs:
I looked through the Pulsar proxy code, [here](https://github.com/apache/pulsar/blob/master/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java#L466) and see that it doesn't factor in anything related to interfaces :sob:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org