You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/03/28 13:02:50 UTC
[GitHub] [pulsar] chairmank edited a comment on issue #3715: Pulsar
functions worker fails to initialize
chairmank edited a comment on issue #3715: Pulsar functions worker fails to initialize
URL: https://github.com/apache/pulsar/issues/3715#issuecomment-477584106
I see this problem only when triggering offload to tiered storage:
```
14:13:53.419 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [foo/test/persistent/redacted] End Offlo[1825/30696]
60, uuid=dc6c1e97-3e9c-4bbd-aa2a-fba14250eedc
14:13:53.420 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [foo/test/persistent/redacted] Preparing metadata
to offload ledger 161 with uuid a929b551-f701-4219-828b-e22dcf79a0e7
14:13:53.428 [bookkeeper-ml-workers-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [foo/test/persistent/redacted] Metadata prepared for
offload of ledger 161 with uuid a929b551-f701-4219-828b-e22dcf79a0e7
14:13:53.447 [offloader-OrderedScheduler-1-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.ArrayIndexOutOfBoundsException: Index -2 out of bounds for length 3
at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:60) ~[com.google.guava-guava-21.0.jar:?]
at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.sendNextRead(PendingReadOp.java:400) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.read(PendingReadOp.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.PendingReadOp.initiate(PendingReadOp.java:526) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.PendingReadOp.safeRun(PendingReadOp.java:536) ~[org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.LedgerHandle.readEntriesInternalAsync(LedgerHandle.java:894) [org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.client.LedgerHandle.readAsync(LedgerHandle.java:757) [org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:4.9.0]
at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlockAwareSegmentInputStreamImpl.readNextEntriesFromLedger(BlockAwareSegmentInputStreamImpl.java:117) [tiered-storage-jc
loud-2.4.0-SNAPSHOT.nar-unpacked/:?]
at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlockAwareSegmentInputStreamImpl.readEntries(BlockAwareSegmentInputStreamImpl.java:89) [tiered-storage-jcloud-2.4.0-SNAP
SHOT.nar-unpacked/:?]
at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlockAwareSegmentInputStreamImpl.read(BlockAwareSegmentInputStreamImpl.java:157) [tiered-storage-jcloud-2.4.0-SNAPSHOT.n
ar-unpacked/:?]
at java.io.InputStream.read(InputStream.java:280) [?:?]
at java.io.InputStream.read(InputStream.java:205) [?:?]
at org.apache.pulsar.shaded.com.google.common.io.ByteStreams.copy(ByteStreams.java:106) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:298) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-
SNAPSHOT]
at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:171) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:65) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at com.sun.proxy.$Proxy72.uploadPart(Unknown Source) [?:?]
at org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:393) [jclouds-shaded-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$offload$2(BlobStoreManagedLedgerOffloader.java:424) [tiered-storage-jcloud-2.4.0-
SNAPSHOT.nar-unpacked/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [com.google.guava-guav
a-21.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [com.google.guava-guava-21.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
```
On https://github.com/apache/bookkeeper/issues/1970, I described my investigation of this issue. Something (I haven't traced exactly what) is calling a method with an entry ID of `-2`, and `org.apache.bookkeeper.client.RoundRobinDistributionSchedule` can not handle a negative `entryId`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services