You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "Sai Sandeep (Jira)" <ji...@apache.org> on 2022/10/14 10:58:00 UTC
[jira] [Updated] (RANGER-3947) Thread leak because of retry loop in SolrCollectionBootstrapper
[ https://issues.apache.org/jira/browse/RANGER-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Sandeep updated RANGER-3947:
--------------------------------
Description:
SolrCollectionBootstrapper Tries to connect to solr cloud and tries upload ranger_audits config and create ranger_audits collection in a loop. We found that in our case uploadConfiguration keeps on failing and because of that is stuck in retry loop.
but because of that it calls connect in a loop which creates new solr cloud client every time. but this solr cloud client is never closed so related connection pools and zk connections are also never closed. because of this number of threads keeps increasing until it crashes.
Related code:
{code:java}
while (!is_completed && (max_retry == TRY_UNTIL_SUCCESS || retry_counter < max_retry)) {
try {
if (connect(zookeeperHosts)) {
if (solr_cloud_mode) {
if (uploadConfiguration() && createCollection()
) {
is_completed = true;
break;
} else {
logErrorMessageAndWait(
"Error while performing operations on solr. ",
null);
}
} } else {
logErrorMessageAndWait(
"Cannot connect to solr kindly check the solr related configs. ",
null);
}
} catch (Exception ex) {
logErrorMessageAndWait("Error while configuring solr. ", ex);
}
}{code}
{code:java}
private boolean connect(List<String> zookeeperHosts) {
try {
logger.info("Solr is in Cloud mode");
if (isKERBEROS) {
setHttpClientBuilderForKrb();
}
solrCloudClient = new CloudSolrClient.Builder(zookeeperHosts,
Optional.empty()).build();
solrCloudClient.setDefaultCollection(solr_collection_name);
solrCloudClient.connect();
zkClient = solrCloudClient.getZkStateReader().getZkClient();
solrClient = solrCloudClient;
solr_cloud_mode = true; return true;
} catch (Exception ex) {
logger.severe("Can't connect to Solr server. ZooKeepers="
+ zookeeperHosts + ", collection=" + solr_collection_name
+ ex);
return false;
}
} {code}
I think if we close the solrCloudClient in the loop then it should fix the bug
was:
SolrCollectionBootstrapper Tries to connect to solr cloud and tries upload ranger_audits config and create ranger_audits collection in a loop. We found that in our case uploadConfiguration keeps on failing and because of that is stuck in retry loop.
but because of that it calls connect in a loop which creates new solr cloud client every time. but this solr cloud client is never closed so related connection pools and zk connections are also never closed. because of this number of threads keeps increasing until it crashes.
Related code:
{code:java}
while (!is_completed && (max_retry == TRY_UNTIL_SUCCESS || retry_counter < max_retry)) {
try {
if (connect(zookeeperHosts)) {
if (solr_cloud_mode) {
if (uploadConfiguration() && createCollection()
) {
is_completed = true;
break;
} else {
logErrorMessageAndWait(
"Error while performing operations on solr. ",
null);
}
} } else {
logErrorMessageAndWait(
"Cannot connect to solr kindly check the solr related configs. ",
null);
}
} catch (Exception ex) {
logErrorMessageAndWait("Error while configuring solr. ", ex);
}
}{code}
I think if we close the solrCloudClient in the loop then it should fix the bug
> Thread leak because of retry loop in SolrCollectionBootstrapper
> ---------------------------------------------------------------
>
> Key: RANGER-3947
> URL: https://issues.apache.org/jira/browse/RANGER-3947
> Project: Ranger
> Issue Type: Bug
> Components: admin
> Affects Versions: 2.3.0
> Reporter: Sai Sandeep
> Priority: Major
>
> SolrCollectionBootstrapper Tries to connect to solr cloud and tries upload ranger_audits config and create ranger_audits collection in a loop. We found that in our case uploadConfiguration keeps on failing and because of that is stuck in retry loop.
> but because of that it calls connect in a loop which creates new solr cloud client every time. but this solr cloud client is never closed so related connection pools and zk connections are also never closed. because of this number of threads keeps increasing until it crashes.
> Related code:
> {code:java}
> while (!is_completed && (max_retry == TRY_UNTIL_SUCCESS || retry_counter < max_retry)) {
> try {
> if (connect(zookeeperHosts)) {
> if (solr_cloud_mode) {
> if (uploadConfiguration() && createCollection()
> ) {
> is_completed = true;
> break;
> } else {
> logErrorMessageAndWait(
> "Error while performing operations on solr. ",
> null);
> }
> } } else {
> logErrorMessageAndWait(
> "Cannot connect to solr kindly check the solr related configs. ",
> null);
> }
> } catch (Exception ex) {
> logErrorMessageAndWait("Error while configuring solr. ", ex);
> }
> }{code}
> {code:java}
> private boolean connect(List<String> zookeeperHosts) {
> try {
> logger.info("Solr is in Cloud mode");
> if (isKERBEROS) {
> setHttpClientBuilderForKrb();
> }
> solrCloudClient = new CloudSolrClient.Builder(zookeeperHosts,
> Optional.empty()).build();
> solrCloudClient.setDefaultCollection(solr_collection_name);
> solrCloudClient.connect();
> zkClient = solrCloudClient.getZkStateReader().getZkClient();
> solrClient = solrCloudClient;
> solr_cloud_mode = true; return true;
> } catch (Exception ex) {
> logger.severe("Can't connect to Solr server. ZooKeepers="
> + zookeeperHosts + ", collection=" + solr_collection_name
> + ex);
> return false;
> }
> } {code}
> I think if we close the solrCloudClient in the loop then it should fix the bug
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)