You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/09/30 16:59:00 UTC
[GitHub] [pinot] richardstartin opened a new issue #7502: Request tracing prevents query completion
richardstartin opened a new issue #7502:
URL: https://github.com/apache/pinot/issues/7502
There are currently a few bugs in the tracing implementation:
1. `TraceContext` accessed concurrently from different threads leading to `ConcurrentModificationException`
```
2021/09/30 17:47:38.875 ERROR [QueryScheduler] [pqr-12] Encountered exception while processing requestId 7 from broker Broker_192.168.0.22_8000
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) ~[?:?]
at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
at org.apache.pinot.core.util.trace.TraceContext$Trace.toJson(TraceContext.java:90) ~[classes/:?]
at org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:189) ~[classes/:?]
at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236) ~[classes/:?]
at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [guava-20.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [guava-20.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [guava-20.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
<img width="1065" alt="Screenshot 2021-09-30 at 17 47 46" src="https://user-images.githubusercontent.com/16439049/135497563-af35fcba-046b-4fce-aa7d-175cc307b3ea.png">
2. `NullPointerException` in query runner thread pool after the request id has been cleared
```
2021/09/30 17:47:48.858 ERROR [QueryScheduler] [pqr-15] Encountered exception while processing requestId 8 from broker Broker_192.168.0.22_8000
java.lang.NullPointerException: null
at org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:188) ~[classes/:?]
at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236) ~[classes/:?]
at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [guava-20.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [guava-20.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [guava-20.0.jar:?]
```
<img width="1018" alt="Screenshot 2021-09-30 at 17 47 53" src="https://user-images.githubusercontent.com/16439049/135498035-8a32eba0-e54b-46b4-a979-e6a736c2f81a.png">
3. Tracing errors cause the query to fail (tracing errors should never impact what's being traced, it should simply be recorded that tracing failed without changing behaviour.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org