You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shenyu0127 (via GitHub)" <gi...@apache.org> on 2023/07/13 17:00:39 UTC
[GitHub] [pinot] shenyu0127 opened a new issue, #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
shenyu0127 opened a new issue, #11099:
URL: https://github.com/apache/pinot/issues/11099
Error: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 78.254 s <<< FAILURE! - in org.apache.pinot.integration.tests.OfflineClusterMemBasedBrokerQueryKillingTest
Error: org.apache.pinot.integration.tests.OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries Time elapsed: 33.108 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at org.apache.pinot.integration.tests.OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries(OfflineClusterMemBasedBrokerQueryKillingTest.java:246)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:744)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
at org.testng.TestNG.runSuites(TestNG.java:1144)
at org.testng.TestNG.run(TestNG.java:1115)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] shenyu0127 commented on issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on issue #11099:
URL: https://github.com/apache/pinot/issues/11099#issuecomment-1634714998
Please assign the issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] shenyu0127 commented on issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on issue #11099:
URL: https://github.com/apache/pinot/issues/11099#issuecomment-1661325913
The test is still flaky.
In https://github.com/apache/pinot/pull/11121, I tried to fix the test by
- reducing the data size so the servers do not go OOM.
- reducing the RAM threshold for killing a thread so that we can still kill a RAM expensive thread
Now the problem is if the RAM threshold is low, it might kill the broker's main thread for the query [1], then the test hangs for ever.
I tried to increase the RAM threshold, the problem is sometimes no broker sub thread gets killed and the test fails. It is hard to control the data size and RAM sampling to ensure we always kill a broker sub thread but not the broker main thread.
Killing a broker main thread is also an expected behavior but it is hard to assert that in a test (the exception thrown when the main thread is killed is [2]).
[1]
```
2023-07-31T11:11:12.0617231Z 11:11:11.877 INFO [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [brw-0] Adding thread to _threadLocalEntry: brw-0
2023-07-31T11:11:12.0618241Z 11:11:11.878 WARN [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Heap used bytes 2045771776 exceeds critical level 1717986918
2023-07-31T11:11:12.0631653Z 11:11:11.879 ERROR [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Query 729626673000000002 got picked because using 11104832 bytes of memory, actual kill committed true}
2023-07-31T11:11:12.0637965Z 11:11:11.879 ERROR [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Current task status recorded is {Thread[jersey-server-managed-async-executor-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-0,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-2,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[jersey-server-managed-async-executor-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-0,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-1,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=nu
ll}, Thread[pqw-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-1,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-2,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[brw-0,7,main]=ThreadEntry{_currentThreadTaskStatus=TaskEntry{_queryId='729626673000000002', _taskId=0, _rootThread=Thread[jersey-server-managed-async-executor-1,5,main]}, _errorStatus=null}, Thread[pqr-1,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-2,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-0,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}}
2023-07-31T11:11:12.0644004Z 11:11:11.976 WARN [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Query aggregation results {729626673000000002=AggregatedStats{_queryId='729626673000000002', _anchorThread=Thread[jersey-server-managed-async-executor-1,5,main], _isAnchorThread=true, _exceptionAtomicReference=null, _allocatedBytes=11104832, _cpuNS=0}} for the previous kill.
2023-07-31T11:11:12.0645488Z 11:11:11.997 INFO [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [pqw-3] Adding thread to _threadLocalEntry: pqw-3
2023-07-31T11:11:12.0646521Z 11:11:12.005 INFO [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [pqw-3] Adding thread to _threadLocalEntry: pqw-3
2023-07-31T11:11:12.0647593Z 11:11:12.006 WARN [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Heap used bytes 1965031424 exceeds critical level 1717986918
2023-07-31T11:11:12.0648634Z 11:11:12.006 ERROR [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Query 729626673000000002 got picked because using 11104832 bytes of memory, actual kill committed true}
2023-07-31T11:11:12.0656280Z 11:11:12.006 ERROR [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Current task status recorded is {Thread[jersey-server-managed-async-executor-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-0,7,main]=ThreadEntry{_currentThreadTaskStatus=TaskEntry{_queryId='Broker_10.1.0.58_18099_729626673000000003_O', _taskId=-1, _rootThread=Thread[pqr-0,7,main]}, _errorStatus=null}, Thread[pqw-2,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-3,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[jersey-server-managed-async-executor-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-1,5,main]=ThreadEntry{_currentThreadTaskStatu
s=idle, _errorStatus=null}, Thread[pqr-0,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-1,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-0,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-1,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-2,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[brw-0,7,main]=ThreadEntry{_currentThreadTaskStatus=TaskEntry{_queryId='729626673000000002', _taskId=0, _rootThread=Thread[jersey-server-managed-async-executor-1,5,main]}, _errorStatus=java.lang.RuntimeException: Query 729626673000000002 got killed because using 11104832 bytes of memory on BROKER: Broker_10.1.0.58_18099, exceeding the quota}, Thread[pqr-1,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-2,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-3,5,main]=ThreadEn
try{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqr-0,7,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}, Thread[pqw-1,5,main]=ThreadEntry{_currentThreadTaskStatus=idle, _errorStatus=null}}
2023-07-31T11:11:12.0663482Z 11:11:12.007 WARN [PerQueryCPUMemAccountantFactory$PerQueryCPUMemResourceUsageAccountant] [CPUMemThreadAccountant] Query aggregation results {729626673000000003=AggregatedStats{_queryId='729626673000000003', _anchorThread=Thread[jersey-server-managed-async-executor-1,5,main], _isAnchorThread=true, _exceptionAtomicReference=null, _allocatedBytes=0, _cpuNS=0}, 729626673000000002=AggregatedStats{_queryId='729626673000000002', _anchorThread=Thread[jersey-server-managed-async-executor-1,5,main], _isAnchorThread=false, _exceptionAtomicReference=java.lang.RuntimeException: Query 729626673000000002 got killed because using 11104832 bytes of memory on BROKER: Broker_10.1.0.58_18099, exceeding the quota, _allocatedBytes=11104832, _cpuNS=0}} for the previous kill.
2023-07-31T11:11:17.8392954Z 11:11:12.006 ERROR [PinotClientRequest] [jersey-server-managed-async-executor-1] Caught exception while processing POST request
2023-07-31T11:11:17.8394641Z java.lang.InterruptedException: null
2023-07-31T11:11:17.8396275Z at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) ~[?:?]
2023-07-31T11:11:17.8398661Z at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369) ~[?:?]
2023-07-31T11:11:17.8529568Z at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278) ~[?:?]
2023-07-31T11:11:17.8530816Z at org.apache.pinot.core.transport.AsyncQueryResponse.getFinalResponses(AsyncQueryResponse.java:86) ~[classes/:?]
2023-07-31T11:11:17.8532048Z at org.apache.pinot.broker.requesthandler.SingleConnectionBrokerRequestHandler.processBrokerRequest(SingleConnectionBrokerRequestHandler.java:115) ~[classes/:?]
2023-07-31T11:11:17.8533325Z at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:701) ~[classes/:?]
2023-07-31T11:11:17.8534409Z at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:260) ~[classes/:?]
2023-07-31T11:11:17.8535545Z at org.apache.pinot.broker.requesthandler.BrokerRequestHandlerDelegate.handleRequest(BrokerRequestHandlerDelegate.java:107) ~[classes/:?]
2023-07-31T11:11:17.8536622Z at org.apache.pinot.broker.api.resources.PinotClientRequest.executeSqlQuery(PinotClientRequest.java:243) ~[classes/:?]
2023-07-31T11:11:17.8537641Z at org.apache.pinot.broker.api.resources.PinotClientRequest.processSqlQueryPost(PinotClientRequest.java:156) ~[classes/:?]
2023-07-31T11:11:17.8538497Z at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2023-07-31T11:11:17.8539230Z at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
2023-07-31T11:11:17.8540056Z at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2023-07-31T11:11:17.8540721Z at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2023-07-31T11:11:17.8542098Z at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8543895Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8545560Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8547243Z at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8548825Z at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8550132Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8551253Z at org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:390) ~[jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8552269Z at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:830) [jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8553080Z at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [jersey-common-2.39.jar:?]
2023-07-31T11:11:17.8553822Z at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [jersey-common-2.39.jar:?]
2023-07-31T11:11:17.8554582Z at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [jersey-common-2.39.jar:?]
2023-07-31T11:11:17.8555357Z at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [jersey-common-2.39.jar:?]
2023-07-31T11:11:17.8556125Z at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [jersey-common-2.39.jar:?]
2023-07-31T11:11:17.8557022Z at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [jersey-common-2.39.jar:?]
2023-07-31T11:11:17.8557976Z at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:825) [jersey-server-2.39.jar:?]
2023-07-31T11:11:17.8558644Z at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
2023-07-31T11:11:17.8559272Z at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
2023-07-31T11:11:17.8559980Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2023-07-31T11:11:17.8560669Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2023-07-31T11:11:17.8561193Z at java.lang.Thread.run(Thread.java:829) [?:?]
```
[2]
```
java.io.IOException: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get a reason, exception: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
at [Source: (String)""; line: 1, column: 0]" while sending request: http://localhost:18099/query/sql
at org.apache.pinot.controller.helix.ControllerTest.sendPostRequest(ControllerTest.java:795)
at org.apache.pinot.integration.tests.ClusterTest.postQuery(ClusterTest.java:473)
at org.apache.pinot.integration.tests.ClusterTest.postQuery(ClusterTest.java:450)
at org.apache.pinot.integration.tests.OfflineClusterMemBasedBrokerQueryKillingTest.lambda$testDigestOOMMultipleQueries$1(OfflineClusterMemBasedBrokerQueryKillingTest.java:225)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get a reason, exception: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
at [Source: (String)""; line: 1, column: 0]" while sending request: http://localhost:18099/query/sql
at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:448)
at org.apache.pinot.controller.helix.ControllerTest.sendPostRequest(ControllerTest.java:791)
... 8 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] aghilz commented on issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "aghilz (via GitHub)" <gi...@apache.org>.
aghilz commented on issue #11099:
URL: https://github.com/apache/pinot/issues/11099#issuecomment-1636719244
so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11099:
URL: https://github.com/apache/pinot/issues/11099#issuecomment-1684666488
Temporarily disabled the test. Keeping the ticket open
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
URL: https://github.com/apache/pinot/issues/11099
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] shenyu0127 commented on issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on issue #11099:
URL: https://github.com/apache/pinot/issues/11099#issuecomment-1637390205
The `OfflineClusterMemBasedBrokerQueryKillingTest` is meant to test that the broker can kill the RAM expensive GROUP BY query. It is flaky because
- in this test the broker and server runs in the same JVM
- the `PerQueryCPUMemResourceUsageAccountant` is registered for the broker (not the server)
- the server may get an `OutOfMemoryError` exception before responding to the broker.
We can fix the test by
- Decreasing the number of rows needed for the GROUP BY query (so the server will not run out of RAM)
- Decreasing the RAM threshold for expensive query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] shenyu0127 commented on issue #11099: OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQueries is flaky
Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on issue #11099:
URL: https://github.com/apache/pinot/issues/11099#issuecomment-1661328420
@jasperjiaguo
Could you take a look at the flaky test?
If it is hard to fix, how about deleting the test because we have tested the same behavior in `OfflineClusterMemBasedServerQueryKillingTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org