[jira] [Commented] (SOLR-15019) Replica placement API needs a way to fetch existing replica metrics

["Houston Putman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286680#comment-17286680 ] 

Houston Putman commented on SOLR-15019:
---------------------------------------

[~ab], I was looking around the available metrics in the metrics endpoint and I noticed something troubling.
 
This patch added the ability to get System Environment Variables via the metrics handler. In theory this should be similar to retrieving the system properties passed to Solr (which has been available since Solr 8 at least). However, there is a big difference between the two. Users decide which system properties should be passed to Solr, and can restrict them. However this is not the case with environment variables. Those can be in use for other applications or generally not meant to be used by Solr at all.

The biggest issue I have is with sensitive information being accessible via the metrics handler through these envVars. Solr is able to protect well-known system properties by default, because these have defined names that Solr is looking for, such as {{zkDigestPassword}} or {{javax.net.ssl.keyStorePassword}}. However, since Solr isn't reading these values from environment variables, it has no knowledge of the environment variables users may be using to store this information. [The code is merely using|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java#L247-L254] the same {{hiddenSysProps}} used to protect sysProps. However this default list is unchanged and the documentation around metrics wasn't changed to reflect that envVars are also visible now.

Is it absolutely necessary to be able to use Env Vars for autoscaling? If we could rely only on system properties, then we wouldn't have to have this additional vector that people have to lock down. I don't really think it's reasonable to expect users to know all sensitive env vars available for a system, and list them in the metrics config. Those could change during runtime or be unknown to the person running the instance.

> Replica placement API needs a way to fetch existing replica metrics
> -------------------------------------------------------------------
>
>                 Key: SOLR-15019
>                 URL: https://issues.apache.org/jira/browse/SOLR-15019
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>             Fix For: master (9.0)
>
>          Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> Replica placement API was introduced in SOLR-14613. It offers a few sample (and simple) implementations of placement plugins.
> However, this API doesn't offer support for retrieving per-replica metrics, which are required for calculating more realistic placements. For example, when calculating placements for ADDREPLICA on an already existing collection the plugin should know what is the size of replica in order to avoid placing large replicas on nodes with insufficient free disk space.
> After discussing this with [~ilan] we propose the following additions to the API:
> * use the existing {{AttributeFetcher}} interface as a facade for retrieving per-replica values (currently it only retrieves per-node values)
> * add {{ShardValues}} interface to represent strongly-typed API for key metrics, such as replica size, number of docs, number of update and search requests.
> Plugins could then use this API like this:
> {code}
> AttributeFetcher attributeFetcher = ...
> SolrCollection solrCollection = ...
> Set<String> metricNames = ...
> attributeFetcher.requestCollectionMetrics(solrCollection, solrCollection.getShardNames(), metricNames);
> AttributeValues attributeValues = attributeFetcher.fetchAttributes();
> ShardValues shardValues = attributeValues.getShardMetrics(solrCollection.getName(), shardName);
> int sizeInGB = shardValues.getSizeInGB(); // retrieves shard leader metrics
> int replicaSizeInGB = shardValues.getSizeInGB(replica);
> {code} 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org