You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/11 11:51:00 UTC
[GitHub] [pulsar] fracasula edited a comment on issue #7682: Stuck consumer: terminate called after throwing an instance of 'std::bad_alloc'
fracasula edited a comment on issue #7682:
URL: https://github.com/apache/pulsar/issues/7682#issuecomment-671898513
We upgraded this morning to 2.6.0 in production with the latest Pulsar Golang client library (i.e. no cgo). Of course we tested our application on both `testing` and `staging` before going live and we didn't find any compatibility issues. Unfortunately as we deployed the new version the scenario described [above](https://github.com/apache/pulsar/issues/7682#issue-667034747) happened in a much larger scale affecting several consumers at the same time.
The backlog started to grow quite a bit so we decided to downgrade back to 2.5.1. That helped a bit but some consumers were definitely still stuck. Upgraded again to 2.6.0, this time with the `autoSkipNonRecoverableData` set to `true`. With these settings and jobs that periodically restart our pods in Kubernetes we managed to get it more stable.
Of course we would like to avoid having to kill our pods on a regular basis so it would be nice to find a solution.
When looking at the logs on the client side there's nothing relevant coming from the Pulsar library, it just reports logs like:
```json
{"level":"debug","name":"cloud-nodes-service-3681597d6aa5773d","severity":"DEBUG","subscription":"cloud-nodes-service","time":"2020-08-11T07:34:27Z","topic":"persistent://public/default/AlarmEventsUpdated"}
{"cnx":"10.1.15.135:40246 -\u003e 10.2.7.212:6650","level":"debug","severity":"DEBUG","time":"2020-08-11T07:34:27Z","topic":"persistent://public/default/AlarmEventsUpdated"}
{"cnx":"10.1.15.135:40246 -\u003e 10.2.7.212:6650","level":"info","producer_name":"cloud-nodes-service-3681597d6aa5773d","severity":"INFO","time":"2020-08-11T07:34:27Z","topic":"persistent://public/default/AlarmEventsUpdated"}
```
On the server side (mostly the Pulsar broker) we got plenty of exceptions though, such as:
```
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:35:58.848 [bookkeeper-ml-workers-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/NodeAlarmCountersUpdated] Error opening ledger for reading at position 297169:27536 - org.apache.bookkeeper.mledger.ManagedLedgerException: Unknown exception
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:35:58.848 [bookkeeper-ml-workers-OrderedExecutor-1-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [public/default/persistent/NodeAlarmCountersUpdated][cloud-nodes-service] read failed from ledger at position:297169:27536 : Unknown exception
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:35:58.848 [bookkeeper-ml-workers-OrderedExecutor-1-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - [persistent://public/default/NodeAlarmCountersUpdated / cloud-nodes-service] Error reading entries at 297169:27536 : Unknown exception, Read Type Normal - Retrying to read in 54.321 seconds
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:00.406 [pulsar-io-23-4] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - [persistent://public/default/NodeInfoUpdated / cloud-spaceroom-service] Retrying read operation
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:00.428 [offloader-OrderedScheduler-1-0] ERROR org.jclouds.http.functions.ParseJson - Error parsing input: null
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker java.lang.NullPointerException: null
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:319) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:287) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$ParameterReader.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:275) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:187) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.gson.Gson.fromJson(Gson.java:888) ~[com.google.code.gson-gson-2.8.2.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.gson.Gson.fromJson(Gson.java:853) ~[com.google.code.gson-gson-2.8.2.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.GsonWrapper.fromJson(GsonWrapper.java:55) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:82) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:76) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:61) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:41) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86) ~[com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.sun.proxy.$Proxy122.getObject(Unknown Source) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.getBlob(GoogleCloudStorageBlobStore.java:246) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.blobstore.internal.BaseBlobStore.getBlob(BaseBlobStore.java:217) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.sun.proxy.$Proxy73.getBlob(Unknown Source) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl.open(BlobStoreBackedReadHandleImpl.java:195) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$5(BlobStoreManagedLedgerOffloader.java:556) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) [com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:00.444 [offloader-OrderedScheduler-1-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed readOffloaded:
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker org.jclouds.http.HttpResponseException: Error parsing input: null
```
or
```
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:04.482 [offloader-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed readOffloaded:
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker org.jclouds.http.HttpResponseException: Error parsing input: null
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker {statusCode=200, message=OK, headers={Server=[UploadServer], ETag=[CNqKo4De+OoCEAE=], X-GUploader-UploadID=[AAANsUn4S2D94evffqRwEGbgp4-4afCAlL11_kSwnURQy8ukjkf4B9-9HFkMFNoBy62PYgKa7ZRFwa-aJB73GpNWRYl4g6Uk8w], Vary=[X-Origin, Origin], Date=[Tue, 11 Aug 2020 07:36:04 GMT]}, payload=[content=true, contentMetadata=[cacheControl=private, max-age=0, must-revalidate, no-transform, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=1234, contentMD5=null, contentType=application/json; charset=UTF-8, expires=Tue Aug 11 07:36:04 UTC 2020], written=false, isSensitive=false]}
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:67) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:41) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86) ~[com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.sun.proxy.$Proxy122.getObject(Unknown Source) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.getBlob(GoogleCloudStorageBlobStore.java:246) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.blobstore.internal.BaseBlobStore.getBlob(BaseBlobStore.java:217) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.sun.proxy.$Proxy73.getBlob(Unknown Source) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl.open(BlobStoreBackedReadHandleImpl.java:195) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$5(BlobStoreManagedLedgerOffloader.java:556) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) [com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [com.google.guava-guava-25.1-jre.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker Caused by: java.lang.NullPointerException
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:319) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.NullFilteringTypeAdapterFactories$MapTypeAdapter.read(NullFilteringTypeAdapterFactories.java:287) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$ParameterReader.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:275) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:187) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.gson.Gson.fromJson(Gson.java:888) ~[com.google.code.gson-gson-2.8.2.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at com.google.gson.Gson.fromJson(Gson.java:853) ~[com.google.code.gson-gson-2.8.2.jar:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.json.internal.GsonWrapper.fromJson(GsonWrapper.java:55) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:82) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:76) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:61) ~[?:?]
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker ... 28 more
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:04.495 [bookkeeper-ml-workers-OrderedExecutor-5-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/AlarmEventsUpdated] Error opening ledger for reading at position 297170:27534 - org.apache.bookkeeper.mledger.ManagedLedgerException: Unknown exception
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:04.495 [bookkeeper-ml-workers-OrderedExecutor-5-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [public/default/persistent/AlarmEventsUpdated][cloud-nodes-service] read failed from ledger at position:297170:27534 : Unknown exception
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:04.495 [bookkeeper-ml-workers-OrderedExecutor-5-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - [persistent://public/default/AlarmEventsUpdated / cloud-nodes-service] Error reading entries at 297170:27534 : Unknown exception, Read Type Normal - Retrying to read in 59.128 seconds
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:04.935 [pulsar-io-23-3] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - [persistent://public/default/NodeInfoUpdated / cloud-spaceroom-service] Retrying read operation
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:04.954 [offloader-OrderedScheduler-1-0] ERROR org.jclouds.http.functions.ParseJson - Error parsing input: null
```
```
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:07.742 [pulsar-io-23-1] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - [persistent://public/default/AlarmEventsUpdated / cloud-nodes-service] Retrying read operation
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:07.746 [pulsar-load-manager-3-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to ZooKeeper because maximum change 10.64431369304657% exceeded threshold 10%; time since last report written is 10.0 seconds
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:07.751 [pulsar-ordered-OrderedExecutor-5-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x101c63ce3fa0003 local:/10.1.29.4:52068 remoteserver:pulsar-zookeeper-1.pulsar-zookeeper/10.1.34.231:2181 lastZxid:47245067907 xid:29534 sent:29536 recv:30476 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/10.1.29.4:8080
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker 07:36:07.761 [offloader-OrderedScheduler-0-0] ERROR org.jclouds.http.functions.ParseJson - Error parsing input: null
pulsar-broker-84bf4f59d6-lgzzp pulsar-broker java.lang.NullPointerException: null
```
Here are the [full logs](https://drive.google.com/file/d/1hQdGbVNuNekIPcW1rynF8En8Bmst0HS4/view?usp=sharing).
Any ideas?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org