You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rodolphogarrido (via GitHub)" <gi...@apache.org> on 2023/07/17 17:40:58 UTC
[GitHub] [pinot] rodolphogarrido opened a new issue, #11127: Failed to update the segment lineage during startReplaceSegments
rodolphogarrido opened a new issue, #11127:
URL: https://github.com/apache/pinot/issues/11127
Hello, I would like to upfront thanks anyone who can help.
Currently I'm not able to use Pinot segment merge and roll-up feature, due to the following error (full log bellow):
`Failed to update the segment lineage during startReplaceSegments`
Sometimes one or two merges do occur with success, but the sequent ones fails with the same error.
**Full error log:**
> 2023-07-17 14:18:22 INFO: [HttpServer] Started.
2023-07-17 14:17:18 2023/07/17 17:17:18.900 INFO [StartServiceManagerCommand] [main] Executing command: StartServiceManager -clusterName pinot-s3-example -zkAddress zookeeper:2181 -port -1 -bootstrapServices []
2023-07-17 14:17:18 2023/07/17 17:17:18.901 INFO [StartServiceManagerCommand] [main] Starting a Pinot [SERVICE_MANAGER] at 3.225s since launch
2023-07-17 14:17:18 2023/07/17 17:17:18.911 INFO [StartServiceManagerCommand] [main] Started Pinot [SERVICE_MANAGER] instance [ServiceManager_pinot-controller-0_-1] at 3.236s since launch
2023-07-17 14:17:18 2023/07/17 17:17:18.927 INFO [StartServiceManagerCommand] [main] Starting a Pinot [CONTROLLER] at 3.251s since launch
2023-07-17 14:18:32 2023/07/17 17:18:32.465 INFO [StartServiceManagerCommand] [main] Started Pinot [CONTROLLER] instance [Controller_pinot-controller-0_9000] at 76.853s since launch
2023-07-17 14:20:10 2023/07/17 17:20:10.929 ERROR [PinotHelixResourceManager] [grizzly-http-server-0] Failed to update the segment lineage during startReplaceSegments. (tableName = events_REALTIME, segmentsFrom = [events__5__1__20230717T1642Z, events__5__2__20230717T1654Z, events__5__3__20230717T1654Z, events__5__4__20230717T1654Z, events__1__0__20230717T1637Z, events__1__1__20230717T1654Z, events__5__5__20230717T1654Z, events__1__2__20230717T1654Z], segmentsTo = [merged_5m_1689614408807_0_events_1689612840000_1689612847000_0])
2023-07-17 14:20:10 org.apache.pinot.spi.utils.retry.RetriableOperationException: java.lang.NullPointerException
2023-07-17 14:20:10 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.startReplaceSegments(PinotHelixResourceManager.java:3237) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.startReplaceSegments(PinotSegmentUploadDownloadRestletResource.java:631) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2023-07-17 14:20:10 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2023-07-17 14:20:10 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2023-07-17 14:20:10 at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at java.lang.Thread.run(Thread.java:829) [?:?]
2023-07-17 14:20:10 Caused by: java.lang.NullPointerException
2023-07-17 14:20:10 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.lambda$startReplaceSegments$16(PinotHelixResourceManager.java:3297) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 ... 28 more
2023-07-17 14:20:10 2023/07/17 17:20:10.937 ERROR [PinotSegmentUploadDownloadRestletResource] [grizzly-http-server-0] Failed to update the segment lineage during startReplaceSegments. (tableName = events_REALTIME, segmentsFrom = [events__5__1__20230717T1642Z, events__5__2__20230717T1654Z, events__5__3__20230717T1654Z, events__5__4__20230717T1654Z, events__1__0__20230717T1637Z, events__1__1__20230717T1654Z, events__5__5__20230717T1654Z, events__1__2__20230717T1654Z], segmentsTo = [merged_5m_1689614408807_0_events_1689612840000_1689612847000_0])
2023-07-17 14:20:10 java.lang.RuntimeException: Failed to update the segment lineage during startReplaceSegments. (tableName = events_REALTIME, segmentsFrom = [events__5__1__20230717T1642Z, events__5__2__20230717T1654Z, events__5__3__20230717T1654Z, events__5__4__20230717T1654Z, events__1__0__20230717T1637Z, events__1__1__20230717T1654Z, events__5__5__20230717T1654Z, events__1__2__20230717T1654Z], segmentsTo = [merged_5m_1689614408807_0_events_1689612840000_1689612847000_0])
2023-07-17 14:20:10 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.startReplaceSegments(PinotHelixResourceManager.java:3353) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.startReplaceSegments(PinotSegmentUploadDownloadRestletResource.java:631) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2023-07-17 14:20:10 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2023-07-17 14:20:10 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2023-07-17 14:20:10 at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at java.lang.Thread.run(Thread.java:829) [?:?]
2023-07-17 14:20:10 Caused by: org.apache.pinot.spi.utils.retry.RetriableOperationException: java.lang.NullPointerException
2023-07-17 14:20:10 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.startReplaceSegments(PinotHelixResourceManager.java:3237) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 ... 27 more
2023-07-17 14:20:10 Caused by: java.lang.NullPointerException
2023-07-17 14:20:10 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.lambda$startReplaceSegments$16(PinotHelixResourceManager.java:3297) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.startReplaceSegments(PinotHelixResourceManager.java:3237) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:20:10 ... 27 more
2023-07-17 14:22:15 2023/07/17 17:22:15.732 ERROR [RetentionManager] [pool-11-thread-8] Failed to clean up the segment lineage. (tableName = events_REALTIME)
2023-07-17 14:22:15 org.apache.pinot.spi.utils.retry.RetriableOperationException: java.lang.NullPointerException
2023-07-17 14:22:15 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.manageSegmentLineageCleanupForTable(RetentionManager.java:199) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.processTable(RetentionManager.java:91) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTable(ControllerPeriodicTask.java:143) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTables(ControllerPeriodicTask.java:116) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(ControllerPeriodicTask.java:85) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTaskScheduler.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2023-07-17 14:22:15 at java.lang.Thread.run(Thread.java:829) [?:?]
2023-07-17 14:22:15 Caused by: java.lang.NullPointerException
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.shouldDeleteReplacedSegments(RetentionManager.java:291) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.lambda$manageSegmentLineageCleanupForTable$0(RetentionManager.java:229) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 ... 14 more
2023-07-17 14:22:15 2023/07/17 17:22:15.735 ERROR [ControllerPeriodicTask] [pool-11-thread-8] Caught exception while processing table: events_REALTIME in task: RetentionManager
2023-07-17 14:22:15 java.lang.RuntimeException: Failed to clean up the segment lineage. (tableName = events_REALTIME)
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.manageSegmentLineageCleanupForTable(RetentionManager.java:271) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.processTable(RetentionManager.java:91) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTable(ControllerPeriodicTask.java:143) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTables(ControllerPeriodicTask.java:116) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(ControllerPeriodicTask.java:85) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTaskScheduler.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2023-07-17 14:22:15 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2023-07-17 14:22:15 at java.lang.Thread.run(Thread.java:829) [?:?]
2023-07-17 14:22:15 Caused by: org.apache.pinot.spi.utils.retry.RetriableOperationException: java.lang.NullPointerException
2023-07-17 14:22:15 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.manageSegmentLineageCleanupForTable(RetentionManager.java:199) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 ... 13 more
2023-07-17 14:22:15 Caused by: java.lang.NullPointerException
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.shouldDeleteReplacedSegments(RetentionManager.java:291) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.lambda$manageSegmentLineageCleanupForTable$0(RetentionManager.java:229) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 at org.apache.pinot.controller.helix.core.retention.RetentionManager.manageSegmentLineageCleanupForTable(RetentionManager.java:199) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
2023-07-17 14:22:15 ... 13 more
# Table and Schema
Schema:
> {
"schemaName": "events",
"dimensionFieldSpecs": [
{
"name": "uuid",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "count",
"dataType": "INT"
}
],
"dateTimeFieldSpecs": [{
"name": "ts",
"dataType": "TIMESTAMP",
"format" : "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}]
}
Table:
> {
"tableName": "events",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "ts",
"schemaName": "events",
"replication": "3",
"replicasPerPartition": "3",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "1"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.topic.name": "events",
"stream.kafka.broker.list": "broker0:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "300",
"realtime.segment.flush.threshold.time": "5m",
"realtime.segment.flush.threshold.segment.size": "5M"
}
},
"tenants": {},
"metadata": {},
"task": {
"taskTypeConfigsMap": {
"MergeRollupTask": {
"5m.mergeType": "concat",
"5m.bucketTimePeriod": "5m",
"5m.bufferTimePeriod": "5m",
"1h.mergeType": "concat",
"1h.bucketTimePeriod": "1h",
"1h.bufferTimePeriod": "1h"
}
}
}
}
# General cluster info
Apache Pinot version: 0.12.1
Number of Servers: 3
Number of Controllers: 1
Number of Brokers: 3
Number of Minions: 2
Deep Storage: minIO
Everything is running in docker-compose right now for testing purposes before deployment in k8s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11127: Failed to update the segment lineage during startReplaceSegments
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11127:
URL: https://github.com/apache/pinot/issues/11127#issuecomment-1644660402
cc @snleee @jtao15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] rodolphogarrido commented on issue #11127: Failed to update the segment lineage during startReplaceSegments
Posted by "rodolphogarrido (via GitHub)" <gi...@apache.org>.
rodolphogarrido commented on issue #11127:
URL: https://github.com/apache/pinot/issues/11127#issuecomment-1645466471
Hi @Jackie-Jiang, thank you very much for replying.
After adding the ingestion type as `APPEND`, the error stopped and Pinot was able to merge the segments.
```json
"ingestionConfig": {
"batchIngestionConfig": {
"segmentIngestionType": "APPEND",
"segmentIngestionFrequency": "DAILY"
},
"transformConfigs": []
},
```
It was possible to reproduce the error by removing the configuration above, and "fix" it after putting it back.
Thank you very much for the help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #11127: Failed to update the segment lineage during startReplaceSegments
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #11127: Failed to update the segment lineage during startReplaceSegments
URL: https://github.com/apache/pinot/issues/11127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11127: Failed to update the segment lineage during startReplaceSegments
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11127:
URL: https://github.com/apache/pinot/issues/11127#issuecomment-1646236533
@rodolphogarrido This has been fixed in the latest master, so after picking up the next release, you shouldn't need to add this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11127: Failed to update the segment lineage during startReplaceSegments
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11127:
URL: https://github.com/apache/pinot/issues/11127#issuecomment-1644671261
This might be relevant: #10403
Can you try adding ingestion type (`APPEND`) to the table config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] rodolphogarrido commented on issue #11127: Failed to update the segment lineage during startReplaceSegments
Posted by "rodolphogarrido (via GitHub)" <gi...@apache.org>.
rodolphogarrido commented on issue #11127:
URL: https://github.com/apache/pinot/issues/11127#issuecomment-1646546455
@Jackie-Jiang Thank you very much for helping me and pointing that this issues is already fixed in the next release 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org