You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "saltosaurus (via GitHub)" <gi...@apache.org> on 2023/05/03 14:22:17 UTC
[GitHub] [pinot] saltosaurus opened a new issue, #10714: Memory leak in the JDBC client
saltosaurus opened a new issue, #10714:
URL: https://github.com/apache/pinot/issues/10714
I'm digging into an issue with OOM errors and a memory leak of some kind using the java-client and jdbc connections to Pinot. Eventually our Spring Boot application dies with:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AsyncHttpClient-timer-1820-1"
In digging into this I found https://github.com/AsyncHttpClient/async-http-client/issues/1658, the important part (considering I see hundreds of threads marked AsyncHttpClient-timer-X-1) of which is:
AsyncHttpClient-x-x: I/O (twice number of cores)
AsyncHttpClient-timer: timer for timeouts (only one)
Any different number means you're creating multiple clients.
I dug into the Pinot code where this issue seems to be coming from and found where it's creating an AsyncHttpClient:
[https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pino[…]/org/apache/pinot/client/JsonAsyncHttpPinotClientTransport.java](https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/JsonAsyncHttpPinotClientTransport.java#L69)
[https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pino[…]rg/apache/pinot/client/controller/PinotControllerTransport.java](https://github.com/apache/pinot/blob/3e1aba37deee21ccb2e764afce789e463098246f/pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/controller/PinotControllerTransport.java#L70)
https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/BrokerCache.java#L120
The problem here as I see it is that the docs for AsyncHttpClient say
`AsyncHttpClient instances are intended to be global resources that share the same lifecycle as the application. Typically, AHC will usually underperform if you create a new client for each request, as it will create new threads and connection pools for each.`
which the Pinot java client is clearly not doing, and looking at the classes that create these client it's up to other code that's using them to close them. I think this is resulting in client (and their respective threadpools) being kept around in parallel causing memory leakage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10714: Memory leak in the JDBC client
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10714:
URL: https://github.com/apache/pinot/issues/10714#issuecomment-1560212628
@KKcorps Can you help take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saltosaurus commented on issue #10714: Memory leak in the JDBC client
Posted by "saltosaurus (via GitHub)" <gi...@apache.org>.
saltosaurus commented on issue #10714:
URL: https://github.com/apache/pinot/issues/10714#issuecomment-1533125747
Our configuration:
`public DataSource pinotDatabase(PinotConfig pinotConfig) {
Map<String, PinotClientProperties> clients = pinotConfig.getClients();
Optional<PinotClientProperties> client = clients.values().stream().findFirst();
String host;
Properties props = new Properties();
props.put("headers.Authorization", toBasicAuthToken(pinotConfig.getUsername(), pinotConfig.getPassword()));
if (client.isPresent()) {
PinotClientProperties clientProperties = client.get();
host = String.format("jdbc:pinot://%s:%s", clientProperties.getHosts().get(0), client.get().getPort());
} else {
throw new DataLabException(false,
"Unable to find cluster name configured under key 'pinot.clients' in application.yml.");
}
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setUrl(host);
dataSource.setDriverClassName("org.apache.pinot.client.PinotDriver");
dataSource.setConnectionProperties(props);
return dataSource;
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saltosaurus commented on issue #10714: Memory leak in the JDBC client
Posted by "saltosaurus (via GitHub)" <gi...@apache.org>.
saltosaurus commented on issue #10714:
URL: https://github.com/apache/pinot/issues/10714#issuecomment-1533124983
Since reporting this issue to Slack I have since copied the [JDBC Driver](https://github.com/apache/pinot/blob/55db06fab16682fcac40d12cbdb7fc1df3ca6a91/pinot-clients/pinot-jdbc-client/src/main/java/org/apache/pinot/client/PinotDriver.java) into our repository and modified it to only keep a single static copy of the AsyncHttpClient.
This has resulted in a significant reduction in leaked memory, though the issue does not seem 100% solved.
See https://apache-pinot.slack.com/archives/C011C9JHN7R/p1681761095987249 for the original Slack thread on this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org